Dashboard > App Engine Pipefitters > Home > Datastore Issues
  App Engine Pipefitters Log In | Sign Up   View a printable version of the current page.  
  Datastore Issues
Added by Mike Orr, last edited by Mike Orr on Jun 26, 2008
Labels: 
(None)

App Engine forces you to use Datastore or an external web service for your database. This is either a severe limitation or an opportunity for a paradigm shift, depending on your viewpoint.

Why no SQLAlchemy?

You can't run a relational database on the App Engine server. You cannot communicate with an external database except via a web service. It would be possible to put a web service front end over DBAPI, and build a SQLAlchemy engine that uses it, but this is a big task.

Running an RDBMS like this would cause the "database bottleneck" common to many large web applications. Web servers can be replicated, and read-only databases can be replicated, but intensive read-write-requery databases are singletons unless you can arrange a master-slave or peer-update arrangement. If demand for your application scales to the level where GAE is necessary, it will overwhelm a relational database, not only in the number of queries per second but in the number of records that would have to be queried.

Datastore query limitations and workarounds

Queries can't return or even count more than 1000 records. All queries return the entire record, even if it contains large chunks of text that will never be accessed during this request. You have to bring a record into memory before deleting it. (XXX There may be a function to bypass this?) Cross-table queries (joins) are not allowed. Several kinds of complex queries are disallowed, such as more than one "!=" operator in the query. Selecting by substring is cumbersome. The only aggregate function is count(), and it won't count more than 1000 records.

Again, these constraints are dictated by the huge scale of Google's apps and the kinds of applications App Engine is designed for. The solution is to keep a minimal number of statistics, and to update them during the request. So if you're counting the number of People records, store a count elsewhere in the database and increment it whenever you add a record.

Site running on a free Atlassian Confluence Open Source Project License granted to Pylons. Evaluate Confluence today.
Powered by Atlassian Confluence, the Enterprise Wiki. (Version: 2.3.3 Build:#645 Feb 13, 2007) - Bug/feature request - Contact Administrators