btree
ZODB3 3.9.0 Beta 1
A free set of tools for using the Zope Object Database (ZODB) more>> A free set of tools for using the Zope Object Database (ZODB)
The Zope Object Database provides an object-oriented database for Python that provides a high-degree of transparency.
Applications can take advantage of object database features with few, if any, changes to application logic. ZODB includes features such as a plugable storage interface, rich transaction support, and undo.
The components you get with the ZODB release are as follows:
- Core ZODB, including the persistence machinery
- Standard storages such as FileStorage
- The persistent BTrees modules
- ZEO, for scalability needs
- documentation (needs more work)
ZODBs primary development platforms are Mac OS X, Linux, and Windows XP.
System requirements:
-
Enhancements
New Features (in more or less reverse chronological order):
- The Database class now has an xrefs keyword argument and a corresponding allow-implicit-cross-references configuration option. which default to true. When set to false, cross-database references are disallowed.
- As a convenience, the connection root method for returning teh root object can now also be used as an object with attributes mapped to the root-object keys.
- Databases have a new method, transaction, that can be used with the Python (2.5 and later) with statement
- This uses a private transaction manager for the connection. If control exists the block without an error, the transaction is committed, otherwise, it is aborted.
- Convenience functions ZODB.connection and ZEO.connection provide a convenient way to open a connection to a database. They open a database and return a connection to it. When the connection is closed, the database is closed as well.
- The ZODB.config databaseFrom... methods now support multi-databases. If multiple zodb sections are used to define multiple databases, the databases are connected in a multi-database arrangement and the first of the defined databases is returned.
The zeopack script has gotten a number of improvements:
- Simplified command-line interface. (The old interface is still supported, except that support for ZEO version 1 servers has been dropped.)
- Multiple storages can be packed in sequence.
- This simplifies pack scheduling on servers serving multiple databases.
- All storages are packed to the same time.
- You can now specify a time of day to pack to.
- The script will now time out if it cant connect to s storage in 60 seconds.
- The connection now estimates the object size based on its pickle size and informs the cache about size changes.
- The database got additional configurations options (cache-size-bytes and historical-cache-size-bytes) to limit the cache size based on the estimated total size of cached objects. The default values are 0 which has the interpretation "do not limit based on the total estimated size". There are corresponding methods to read and set the new configuration parameters.
- Connections now have a public opened attribute that is true when the connection is open, and false otherwise. When true, it is the seconds since the epoch (time.time()) when the connection was opened. This is a renaming of the previous _opened private variable.
- FileStorage now supports blobs directly.
- You can now control whether FileStorages keep .old files when packing.
- POSKeyErrors are no longer logged by ZEO servers, because they are really client errors.
- A new storage interface, IExternalGC, to support external garbage collection, http://wiki.zope.org/ZODB/ExternalGC, has been defined and implemented for FileStorage and ClientStorage.
- As a small convenience (mainly for tests), you can now specify initial data as a string argument to the Blob constructor.
- ZEO Servers now provide an option, invalidation-age, that allows quick verification of ZEO clients less than a given age even if the number of transactions the client hasnt seen exceeds the invalidation queue size. This is only recommended if the storage being served supports effecient iteration from a point near the end of the transaction history.
- The FileStorage iterator now handles large files better. When iteratng from a starting transaction near the end of the file, the iterator will scan backward from the end of the file to find the starting point. This enhancement makes it practical to take advantage of the new storage server invalidation-age option.
- Previously, database connections were managed as a stack. This tended to cause the same connection(s) to be used over and over. For example, the most used conection would typically be the onlyt connection used. In some rare situations, extra connections could be opened and end up on the top of the stack, causing extreme memory wastage. Now, when connections are placed on the stack, they sink below existing connections that have more active objects.
- There is a new pool-timeout database configuration option to specify that connections unused after the given time interval should be garbage colection. This will provide a means of dealing with extra connections that are created in rare circumstances and that would consume an unreasonable amount of memory.
- The Blob open method now supports a new mode, c, to open committed data for reading as an ordinary file, rather than as a blob file. The ordinary file may be used outside the current transaction and even after the blobs database connection has been closed.
- ClientStorage now provides blob cache management. When using non-shared blob directories, you can set a target cache size and the cache will periodically be reduced try to keep it below the target size.
- The client blob directory layout has changed. If you have existing non-shared blob directories, you will have to remove them.
- ZODB 3.9 ZEO clients can connect to ZODB 3.8 servers. ZODB ZEO clients from ZODB 3.2 on can connect to ZODB 3.9 servers.
- When a ZEO cache is stale and would need verification, a ZEO.interfaces.StaleCache event is published (to zope.event). Applications may handle this event and take action such as exiting the application without verifying the cache or starting cold.
- Theres a new convenience function, ZEO.DB, for creating databases using ZEO Client Storages. Just call ZEO.DB with the same arguments you would otherwise pass to ZEO.ClientStorage.ClientStorage.
- Object saves are a little faster
- When configuring storages in a storage server, the storage name now defaults to "1". In the overwhelmingly common case that a single storage, the name can now be ommitted.
- FileStorage now provides optional garbage collection. A gc keyword option can be passed to the pack method. A false value prevents garbage collection.
- The FileStorage constructor now provides a boolean pack_gc option, which defaults to True, to control whether garbage collection is performed when packing by default. This can be overridden with the gc option to the pack method.
- The ZConfig configuration for FileStorage now includes a pack-gc option, corresponding to the pack_gc constructor argument.
- The FileStorage constructor now has a packer keyword argument that allows an alternative packer to be supplied.
- The ZConfig configuration for FileStorage now includes a packer option, corresponding to the packer constructor argument.
- MappingStorage now supports multi-version concurrency control and iteration and provides a better storage implementation example.
DemoStorage has a number of new features:
- The ability to use a separate storage, such as a file storage to store changes
- Blob support
- Multi-version concurrency control and iteration
- Explicit support dfor demo-storage stacking via push and pop methods.
- Wen calling ZODB.DB to create a database, you can now pass a file name, rather than a storage to use a file storage.
Added support for copying and recovery of blob storages:
- Added a helper function, ZODB.blob.is_blob_record for testing whether a data record is for a blob. This can be used when iterating over a storage to detect blob records so that blob data can be copied. In the future, we may want to build this into a blob-aware iteration interface, so that records get blob file attributes automatically.
- Added the IBlobStorageRestoreable interfaces for blob storages that support recovery via a restoreBlob method.
- Updated ZODB.blob.BlobStorage to implement IBlobStorageRestoreable and to have a copyTransactionsFrom method that also copies blob data.
- New ClientStorage configuration option drop_cache_rather_verify. If this option is true then the ZEO client cache is dropped instead of the long (unoptimized) verification. For large caches, setting this option can avoid effective downtimes in the order of hours when the connection to the ZEO server was interrupted for a longer time.
- Cleaned-up the storage iteration API and provided an iterator implementation for ZEO.
- Versions are no-longer supported.
- Document conflict resolution (see ZODB/ConflictResolution.txt).
- Support multidatabase references in conflict resolution.
- Make it possible to examine oid and (in some situations) database name of persistent object references during conflict resolution.
- Moved the transaction module out of ZODB. ZODB depends upon this module, but it must be installed separately.
- ZODB installation now requires setuptools.
- Added offset information to output of fstail script. Added test harness for this script.
- Added support for read-only, historical connections based on datetimes or serials (TIDs). See src/ZODB/historical_connections.txt.
- Removed the ThreadedAsync module.
- Now depend on zc.lockfile
Bugs Fixed:
- fixed Python 2.6 compatibility issue with ZEO/zeoserverlog.py
- using hashlib.sha1 if available in order to avoid DeprecationWarning under Python 2.6
- made runzeo -h work
- The monitor server didnt correctly report the actual number of clients.
- Packing could return spurious errors due to errors notifying disconnected clients of new database size statistics.
- Undo sometimes failed for FileStorages configured to support blobs.
- Starting ClientStorages sometimes failed with non-new but empty cache files.
- The history method on ZEO clients failed.
- Fix for bug #251037: Make packing of blob storages non-blocking.
- Fix for bug #220856: Completed implementation of ZEO authentication.
- Fix for bug #184057: Make initialisation of small ZEO client file cache sizes not fail.
- Fix for bug #184054: MappingStorage used to raise a KeyError during load instead of a POSKeyError.
- Fixed bug in Connection.TmpStore: load() would not defer to the backend storage for loading blobs.
- Fix for bug #181712: Make ClientStorage update lastTransaction directly after connecting to a server, even when no cache verification is necessary.
- Fixed bug in blob filesystem helper: the isSecure check was inversed.
- Fixed bug in transaction buffer: a tuple was unpacked incorrectly in clear.
- Bugfix the situation in which comparing persistent objects (for instance, as members in BTree set or keys of BTree) might cause data inconsistency during conflict resolution.
- Fixed bug 153316: persistent and BTrees were using int for memory sizes which caused errors on x86_64 Intel Xeon machines (using 64-bit Linux).
- Fixed small bug that the Connection.isReadOnly method didnt work after a savepoint.
- Bug #98275: Made ZEO cache more tolerant when invalidating current versions of objects.
- Fixed a serious bug that could cause client I/O to stop (hang). This was accomonied by a critical log message along the lines of: "RuntimeError: dictionary changed size during iteration".
- Fixed bug #127182: Blobs were subclassable which was not desired.
- Fixed bug #126007: tpc_abort had untested code path that was broken.
- Fixed bug #129921: getSize() function in BlobStorage could not deal with garbage files
- Fixed bug in which MVCC would not work for blobs.
- Fixed bug in ClientCache that occurred with objects larger than the total cache size.
- When an error occured attempting to lock a file and logging of said error was enabled.
- FileStorages previously saved indexes after a certain number of writes. This was done during the last phase of two-phase commit, which made this critical phase more subject to errors than it should have been. Also, for large databases, saves were done so infrequently as to be useless. The feature was removed to reduce the chance for errors during the last phase of two-phase commit.
- File storages previously kept an internal object id to transaction id mapping as an optimization. This mapping caused excessive memory usage and failures during the last phase of two-phase commit. This optimization has been removed.
- Refactored handling of invalidations on ZEO clients to fix a possible ordering problem for invalidation messages.
- On many systems, it was impossible to create more than 32K blobs. Added a new blob-directory layout to work around this limitation.
- Fixed bug that could lead to memory errors due to the use of a Python dictionary for a mapping that can grow large.
- Fixed bug #251037: Made packing of blob storages non-blocking.
- Fixed a bug that could cause InvalidObjectReference errors for objects that were explicitly added to a database if the object was modified after a savepoint that added the object.
- Fixed several bugs that caused ZEO cache corruption when connecting to servers. These bugs affected both persistent and non-persistent caches.
- Improved the the ZEO client shutdown support to try to avoid spurious errors on exit, especially for scripts, such as zeopack.
- Packing failed for databases containing cross-database references.
- Cross-database references to databases with empty names werent constructed properly.
- The zeo client cache used an excessive amount of memory, causing applications with large caches to exhaust available memory.
Fixed a number of bugs in the handling of persistent ZEO caches:
- Cache records are written in several steps. If a process exits after writing begins and before it is finishes, the cache will be corrupt on restart. The way records are writted was changed to make cache record updates atomic.
- There was no lock file to prevent opening a cache multiple times at once, which would lead to corruption. Persistent caches now use lock files, in the same way that file storages do.
- A bug in the cache-opening logic led to cache failure in the unlikely event that a cache has no free blocks.
- When using ZEO Client Storages, Errors occured when trying to store objects too big to fit in the ZEO cache file.
- Fixed bug in blob filesystem helper: the isSecure check was inversed.
- Fixed bug in transaction buffer: a tuple was unpacked incorrectly in clear.
- Fixed bug in Connection.TmpStore: load() would not defer to the backend storage for loading blobs.
- Fixed bug #190884: Wrong reference to POSKeyError caused NameError.
- Completed implementation of ZEO authentication. This fixes issue 220856.
Oracle Berkeley DB Java Edition 3.3.82
High performance transactional storage engine. more>> High performance transactional storage engine.
Oracle Berkeley DB Java Edition is a high performance, transactional storage engine written entirely in Java. Like the highly successful Berkeley DB product, Berkeley DB Java Edition executes in the address space of the application, without the overhead of client/server communication.
It stores data in the applications native format, so no runtime data translation is required. Berkeley DB Java Edition supports full ACID transactions and recovery. It provides an easy-to-use, programmatic interface, allowing developers to store and retrieve information quickly, simply and reliably.
Main features:
- Local, in-process data storage.
- Schema-neutral, application native data storage.
- Keyed and sequential data retrieval.
- Easy-to-use Java Collections API.
- Persistence API for accessing Java objects.
- Single process, multi-threading model.
- Record level locking for high concurrency.
- Support for secondary indexes.
- In-memory, on disk or both.
- Configurable background cleaner threads re-organize data and optimize disk use.
- Full ACID compliance.
- Selectable isolation levels and durability guarantees, configurable on a per-transaction basis.
- Managed transactions using the Java Transaction API (JTA).
- J2EE application server integration using J2EE Connector Architecture(JCA).
- Provides support for monitoring, auditing and administration using the Java Management Extensions (JMX).
- Catastrophic and routine failure recovery modes.
- Timeout based deadlock detection.
- Hot and cold backups, log file compaction, and full database dumps.
- 100% pure Java for portability and ease of development.
- Single JAR file - easy to install, runs in the same JVM as the application.
- Java 1.4.2 or later Standard Edition JVM required.
- Programmatic administration and management-zero human administration.
- API for routine administrative functions.
- Small footprint 772KB.
- Scalable to terabytes of data, millions of records.
- Source code, test suite included.
System requirements:
-
Enhancements
- Fix a bug that somtimes causes two log cleaner threads (when multiple threads are configured or multiple app threads call Environment.cleanLog) to select the same file to clean. This can result in two attempts to delete the log file after cleaning. The second attempt causes an exception to be traced repeatedly, but no other side effect. The problem is transient and goes away if the Environment is closed and re-opened. An example of the exception is below. The symptom is that this exception occurs repeatedly, and the file is not present.
- Fix a bug that could cause a LogFileNotFoundException when using an XAEnvironment, if a prepared transaction is not ended prior to a crash and then the prepared transaction is aborted after recovering from a crash. [#17022] (3.3.79)
- Fix a bug that prevented deferred-write record deletions from being made durable by Database.sync, if a crash occurs after Database.sync but prior to the next checkpoint. Under rare circumstances this could also result in a LogFileNotFoundException later when accessing the deleted entry. [#16864] (3.3.78)
- Fix a bug that prevents log cleaning from functioning properly when a temporary DB (DatabaseConfig.setTemporary) is large enough to overflow the JE cache. Also fix a bug that could in rare cirumstances cause an endless loop while performing log cleaning. [#16928] (3.3.78)
- Fix a bug that caused an infinite loop under certain timing dependent circumstances when using EnvironmentConfig.setSharedCache(true). This bug was reported in two different forum posts. Thanks to Kevin Jackson and Vyacheslav Lukianov for reporting it! [#16882] (3.3.78)
- Improve output of partial information in DatabaseEntry.toString. Thanks to archie172 for reporting this on OTN. [#16929] (3.3.78)
- Fix a bug in SecondaryCursor.dup that required the cursor to be initialized. This method, like Cursor.dup, may now be called whether the cursor is initialized or not. Thanks to archie172 for reporting this on OTN. [#16930] (3.3.78)
- Fix a bug that caused NullPointerException when opening an XAEnvironment under certain circumstances, when prepared but uncommitted transactions are present in the JE log and must be replayed during recovery. This bug was reported on the OTN forum by Matthieu Bentot -- thank you Matthieu! An example of the stack trace is below.
- Fix a bug that caused LogFileNotFoundException in rare circumstances for an Environment having one or more Databases configured for duplicates (or one or more SecondaryIndexes with MANY_TO_XXX relationships). The bug only occurs when a single secondary/duplicate key value is associated with a large number of records/entities; specifically, the sub-Btree for a single duplicate key value must have at least 3 levels. [#16712] (3.3.76)
- Fix a bug that caused BufferOverflowException while writing transactional records. This could occur if multiple threads were writing to an environment while using the same Transaction. [#17204]
- Direct Persistence Layer (DPL), Collections and Bind packages - Fix a bug that causes the EvolveStats returned by EntityStore.evolve to contain zeros, even when entities are evolved and written, when no EvolveListener is configured. Thanks to jhalex on OTN for reporting this. [#17024] (3.3.78)
- Page: 1 of 1
- 1