Technical Overview

Document Storage

Documents are the primary unit of data in CouchDB and consist of any number of fields and attachments. Documents also include metadata that’s maintained by the database system. Document fields are uniquely named and contain values of varying types (text, number, boolean, lists, etc), and there is no set limit to text size or element count.

The CouchDB document update model is lockless and optimistic. Document edits are made by client applications loading documents, applying changes, and saving them back to the database. If another client editing the same document saves their changes first, the client gets an edit conflict error on save. To resolve the update conflict, the latest document version can be opened, the edits reapplied and the update tried again.

Single document updates (add, edit, delete) are all or nothing, either succeeding entirely or failing completely. The database never contains partially saved or edited documents.

ACID Properties

Document updates (add, edit, delete) are serialized, except for binary blobs which are written concurrently. Database readers are never locked out and never have to wait on writers or other readers. Any number of clients can be reading documents without being locked out or interrupted by concurrent updates, even on the same document. CouchDB read operations use a Multi-Version Concurrency Control (MVCC) model where each client sees a consistent snapshot of the database from the beginning to the end of the read operation. This means that CouchDB can guarantee transactional semantics on a per-document basis.

Documents are indexed in B-trees by their name (DocID) and a Sequence ID. Each update to a database instance generates a new sequential number. Sequence IDs are used later for incrementally finding changes in a database. These B-tree indexes are updated simultaneously when documents are saved or deleted. The index updates always occur at the end of the file (append-only updates).

Documents have the advantage of data being already conveniently packaged for storage rather than split out across numerous tables and rows in most database systems. When documents are committed to disk, the document fields and metadata are packed into buffers, sequentially one document after another (helpful later for efficient building of views).

When CouchDB documents are updated, all data and associated indexes are flushed to disk and the transactional commit always leaves the database in a completely consistent state. Commits occur in two steps:

In the event of an OS crash or power failure during step 1, the partially flushed updates are simply forgotten on restart. If such a crash happens during step 2 (committing the header), a surviving copy of the previous identical headers will remain, ensuring coherency of all previously committed data. Excepting the header area, consistency checks or fix-ups after a crash or a power failure are never necessary.

Compaction

Views

But beyond acting as a fancy file server, a simple document model for data storage and sharing is too simple to build real applications on – it simply doesn’t do enough of the things we want and expect. We want to slice and dice and see our data in many different ways. What is needed is a way to filter, organize and report on data that hasn’t been decomposed into tables.

SEE ALSO:

View Model

View definitions are strictly virtual and only display the documents from the current database instance, making them separate from the data they display and compatible with replication. CouchDB views are defined inside special design documents and can replicate across database instances like regular documents, so that not only data replicates in CouchDB, but entire application designs replicate too.

JavaScript View Functions

SEE ALSO:

View Indexes

To keep view querying fast, the view engine maintains indexes of its views, and incrementally updates them to reflect changes in the database. CouchDB’s core design is largely optimized around the need for efficient, incremental creation of views and their indexes.

Views and their functions are defined inside special “design” documents, and a design document may contain any number of uniquely named view functions. When a user opens a view and its index is automatically updated, all the views in the same design document are indexed as a single group.

The view builder uses the database sequence ID to determine if the view group is fully up-to-date with the database. If not, the view engine examines all database documents (in packed sequential order) changed since the last refresh. Documents are read in the order they occur in the disk file, reducing the frequency and cost of disk head seeks.

The views can be read and queried simultaneously while also being refreshed. If a client is slowly streaming out the contents of a large view, the same view can be concurrently opened and refreshed for another client without blocking the first client. This is true for any number of simultaneous client readers, who can read and query the view while the index is concurrently being refreshed for other clients without causing problems for the readers.

As documents are processed by the view engine through your ‘map’ and ‘reduce’ functions, their previous row values are removed from the view indexes, if they exist. If the document is selected by a view function, the function results are inserted into the view as a new row.

When view index changes are written to disk, the updates are always appended at the end of the file, serving to both reduce disk head seek times during disk commits and to ensure crashes and power failures can not cause corruption of indexes. If a crash occurs while updating a view index, the incomplete index updates are simply lost and rebuilt incrementally from its previously committed state.

Security and Validation

SEE ALSO:

Administrator Access

Update Validation

Both the user’s credentials and the updated document are given as inputs to the validation formula, and can be used to implement custom security models by validating a user’s permissions to update a document.

A basic “author only” update document model is trivial to implement, where document updates are validated to check if the user is listed in an “author” field in the existing document. More dynamic models are also possible, like checking a separate user account profile for permission settings.

The update validations are enforced for both live usage and replicated updates, ensuring security and data validation in a shared, distributed system.

SEE ALSO:

Distributed Updates and Replication

The CouchDB document storage, view and security models are designed to work together to make true bi-directional replication efficient and reliable. Both documents and designs can replicate, allowing full database applications (including application design, logic and data) to be replicated to laptops for offline use, or replicated to servers in remote offices where slow or unreliable connections make sharing data difficult.

The replication process is incremental. At the database level, replication only examines documents updated since the last replication. If replication fails at any step, due to network problems or crash for example, the next replication restarts at the last checkpoint.

Partial replicas can be created and maintained. Replication can be filtered by a JavaScript function, so that only particular documents or those meeting specific criteria are replicated. This can allow users to take subsets of a large shared database application offline for their own use, while maintaining normal interaction with the application and that subset of data.

Conflicts

CouchDB allows for any number of conflicting documents to exist simultaneously in the database, with each database instance deterministically deciding which document is the “winner” and which are conflicts. Only the winning document can appear in views, while “losing” conflicts are still accessible and remain in the database until deleted or purged during database compaction. Because conflict documents are still regular documents, they replicate just like regular documents and are subject to the same security and validation rules.

When distributed edit conflicts occur, every database replica sees the same winning revision and each has the opportunity to resolve the conflict. Resolving conflicts can be done manually or, depending on the nature of the data and the conflict, by automated agents. The system makes decentralized conflict resolution possible while maintaining single document database semantics.

Conflict management continues to work even if multiple disconnected users or agents attempt to resolve the same conflicts. If resolved conflicts result in more conflicts, the system accommodates them in the same manner, determining the same winner on each machine and maintaining single document semantics.

SEE ALSO:

Applications

With very little database work, it is possible to build a distributed document management application with granular security and full revision histories. Updates to documents can be implemented to exploit incremental field and blob replication, where replicated updates are nearly as efficient and incremental as the actual edit differences (“diffs”).

Implementation

Both in syntax and semantics, Erlang is very different from conventional programming languages like C or Java. Erlang uses lightweight “processes” and message passing for concurrency, it has no shared state threading and all data is immutable. The robust, concurrent nature of Erlang is ideal for a database server.

CouchDB is designed for lock-free concurrency, in the conceptual model and the actual Erlang implementation. Reducing bottlenecks and avoiding locks keeps the entire system working predictably under heavy loads. CouchDB can accommodate many clients replicating changes, opening and updating documents, and querying views whose indexes are simultaneously being refreshed for other clients, without needing locks.

For higher availability and more concurrent users, CouchDB is designed for “shared nothing” clustering. In a “shared nothing” cluster, each machine is independent and replicates data with its cluster mates, allowing individual server failures with zero downtime. And because consistency scans and fix-ups aren’t needed on restart, if the entire cluster fails – due to a power outage in a datacenter, for example – the entire CouchDB distributed system becomes immediately available after a restart.

CouchDB is built from the start with a consistent vision of a distributed document database system. Unlike cumbersome attempts to bolt distributed features on top of the same legacy models and databases, it is the result of careful ground-up design, engineering and integration. The document, view, security and replication models, the special purpose query language, the efficient and robust disk layout and the concurrent and reliable nature of the Erlang platform are all carefully integrated for a reliable and efficient system.

Why CouchDB?

As CouchDB developers, we’re naturally very excited to be using CouchDB. In this topic we’ll share with you the reasons for our enthusiasm. We’ll show you how CouchDB’s schema-free document model is a better fit for common applications, how the built-in query engine is a powerful way to use and process your data, and how CouchDB’s design lends itself to modularization and scalability.

Relax

Apache CouchDB has started. Time to relax.

Why is relaxation important? Developer productivity roughly doubled in the last five years. The chief reason for the boost is more powerful tools that are easier to use. Take Ruby on Rails as an example. It is an infinitely complex framework, but it’s easy to get started with. Rails is a success story because of the core design focus on ease of use. This is one reason why CouchDB is relaxing: learning CouchDB and understanding its core concepts should feel natural to most everybody who has been doing any work on the Web. And it is still pretty easy to explain to non-technical people.

Getting out of the way when creative people try to build specialized solutions is in itself a core feature and one thing that CouchDB aims to get right. We found existing tools too cumbersome to work with during development or in production, and decided to focus on making CouchDB easy, even a pleasure, to use.

Another area of relaxation for CouchDB users is the production setting. If you have a live running application, CouchDB again goes out of its way to avoid troubling you. Its internal architecture is fault-tolerant, and failures occur in a controlled environment and are dealt with gracefully. Single problems do not cascade through an entire server system but stay isolated in single requests.

CouchDB’s core concepts are simple (yet powerful) and well understood. Operations teams (if you have a team; otherwise, that’s you) do not have to fear random behavior and untraceable errors. If anything should go wrong, you can easily find out what the problem is, but these situations are rare.

CouchDB is also designed to handle varying traffic gracefully. For instance, if a website is experiencing a sudden spike in traffic, CouchDB will generally absorb a lot of concurrent requests without falling over. It may take a little more time for each request, but they all get answered. When the spike is over, CouchDB will work with regular speed again.

The third area of relaxation is growing and shrinking the underlying hardware of your application. This is commonly referred to as scaling. CouchDB enforces a set of limits on the programmer. On first look, CouchDB might seem inflexible, but some features are left out by design for the simple reason that if CouchDB supported them, it would allow a programmer to create applications that couldn’t deal with scaling up or down.

NOTE:

A Different Way to Model Your Data

CouchDB’s design borrows heavily from web architecture and the concepts of resources, methods, and representations. It augments this with powerful ways to query, map, combine, and filter your data. Add fault tolerance, extreme scalability, and incremental replication, and CouchDB defines a sweet spot for document databases.

A Better Fit for Common Applications

Self-Contained Data

Syntax and Semantics

Jan’s business card might contain a phone number but no fax number, whereas J. Chris’s business card contains both a phone and a fax number. Jan does not have to make his lack of a fax machine explicit by writing something as ridiculous as “Fax: None” on the business card. Instead, simply omitting a fax number implies that he doesn’t have one.

We can see that real-world documents of the same type, such as business cards, tend to be very similar in semantics – the sort of information they carry, but can vary hugely in syntax, or how that information is structured. As human beings, we’re naturally comfortable dealing with this kind of variation.

While a traditional relational database requires you to model your data up front, CouchDB’s schema-free design unburdens you with a powerful way to aggregate your data after the fact, just like we do with real-world documents. We’ll look in depth at how to design applications with this underlying storage paradigm.

Building Blocks for Larger Systems

Whether you need a system that’s crazy fast but isn’t too concerned with reliability (think logging), or one that guarantees storage in two or more physically separated locations for reliability, but you’re willing to take a performance hit, CouchDB lets you build these systems.

There are a multitude of knobs you could turn to make a system work better in one area, but you’ll affect another area when doing so. One example would be the CAP theorem discussed in intro/consistency. To give you an idea of other things that affect storage systems, see Figure 2 and Figure 3.

By reducing latency for a given system (and that is true not only for storage systems), you affect concurrency and throughput capabilities.

CouchDB Replication

CouchDB replication uses the same REST API all clients use. HTTP is ubiquitous and well understood. Replication works incrementally; that is, if during replication anything goes wrong, like dropping your network connection, it will pick up where it left off the next time it runs. It also only transfers data that is needed to synchronize databases.

A core assumption CouchDB makes is that things can go wrong, like network connection troubles, and it is designed for graceful error recovery instead of assuming all will be well. The replication system’s incremental design shows that best. The ideas behind “things that can go wrong” are embodied in the Fallacies of Distributed Computing:

Existing tools often try to hide the fact that there is a network and that any or all of the previous conditions don’t exist for a particular system. This usually results in fatal error scenarios when something finally goes wrong. In contrast, CouchDB doesn’t try to hide the network; it just handles errors gracefully and lets you know when actions on your end are required.

Local Data Is King

What do you do when you are offline? This happens all the time – your DSL or cable provider has issues, or your iPhone, G1, or Blackberry has no bars, and no connectivity means no way to get to your data.

CouchDB can solve this scenario as well, and this is where scaling is important again. This time it is scaling down. Imagine CouchDB installed on phones and other mobile devices that can synchronize data with centrally hosted CouchDBs when they are on a network. The synchronization is not bound by user interface constraints like sub-second response times. It is easier to tune for high bandwidth and higher latency than for low bandwidth and very low latency. Mobile applications can then use the local CouchDB to fetch data, and since no remote networking is required for that, latency is low by default.

Can you really use CouchDB on a phone? Erlang, CouchDB’s implementation language has been designed to run on embedded devices magnitudes smaller and less powerful than today’s phones.

Wrapping Up

Eventual Consistency

Working with the Grain

Engineering distributed systems is tricky. Many of the caveats and “gotchas” you will face over time aren’t immediately obvious. We don’t have all the solutions, and CouchDB isn’t a panacea, but when you work with CouchDB’s grain rather than against it, the path of least resistance leads you to naturally scalable applications.

Of course, building a distributed system is only the beginning. A website with a database that is available only half the time is next to worthless. Unfortunately, the traditional relational database approach to consistency makes it very easy for application programmers to rely on global state, global clocks, and other high availability no-nos, without even realizing that they’re doing so. Before examining how CouchDB promotes scalability, we’ll look at the constraints faced by a distributed system. After we’ve seen the problems that arise when parts of your application can’t rely on being in constant contact with each other, we’ll see that CouchDB provides an intuitive and useful way for modeling applications around high availability.

The CAP Theorem

The CAP theorem, shown in Figure 1. The CAP theorem, identifies three distinct concerns:

Pick two.

Local Consistency

The Key to Your Data

No Locking

NOTE:

Instead of locks, CouchDB uses Multi-Version Concurrency Control (MVCC) to manage concurrent access to the database. Figure 3. MVCC means no locking illustrates the differences between MVCC and traditional locking mechanisms. MVCC means that CouchDB can run at full speed, all the time, even under high load. Requests are run in parallel, making excellent use of every last drop of processing power your server has to offer.

Validation

CouchDB can validate documents using JavaScript functions similar to those used for MapReduce. Each time you try to modify a document, CouchDB will pass the validation function a copy of the existing document, a copy of the new document, and a collection of additional information, such as user authentication details. The validation function now has the opportunity to approve or deny the update.

By working with the grain and letting CouchDB do this for us, we save ourselves a tremendous amount of CPU cycles that would otherwise have been spent serializing object graphs from SQL, converting them into domain objects, and using those objects to do application-level validation.

Distributed Consistency

Incremental Replication

Need to scale out your CouchDB database cluster? Just throw in another server.

As illustrated in Figure 4. Incremental replication between CouchDB nodes, with CouchDB’s incremental replication, you can synchronize your data between any two databases however you like and whenever you like. After replication, each database is able to work independently.

You could use this feature to synchronize database servers within a cluster or between data centers using a job scheduler such as cron, or you could use it to synchronize data with your laptop for offline work as you travel. Each database can be used in the usual fashion, and changes between databases can be synchronized later in both directions.

Case Study

NOTE:

Let’s examine the workflow of the Songbird backup application, first as a user backing up from a single computer, and then using Songbird to synchronize playlists between multiple computers. We’ll see how document revisions turn what could have been a hairy problem into something that just works.

The first time we use this backup application, we feed our playlists to the application and initiate a backup. Each playlist is converted to a JSON object and handed to a CouchDB database. As illustrated in Figure 5. Backing up to a single database, CouchDB hands back the document ID and revision of each playlist as it’s saved to the database.

Wrapping Up

We’ve covered the main issues surrounding CouchDB’s consistency model and hinted at some of the benefits to be had when you work with CouchDB and not against it. But enough theory – let’s get up and running and see what all the fuss is about!

cURL: Your Command Line Friend

For simple GET requests you can supply the URL of the request. For example, to get the database information:

shell> curl http://admin:password@127.0.0.1:5984

This returns the database information (formatted in the output below for clarity):

{
  "couchdb": "Welcome",
  "version": "3.0.0",
  "git_sha": "83bdcf693",
  "uuid": "56f16e7c93ff4a2dc20eb6acc7000b71",
  "features": [
    "access-ready",
    "partitioned",
    "pluggable-storage-engines",
    "reshard",
    "scheduler"
  ],
  "vendor": {
    "name": "The Apache Software Foundation"
  }
}

NOTE:

shell> curl 'http://couchdb:5984/_uuids?count=5'

NOTE:

shell> curl -X PUT 'http://127.0.0.1:5984/demo/doc' -d '{"motto": "I love gnomes"}'

shell> curl -X PUT "http://127.0.0.1:5984/demo/doc" -d "{""motto"": ""I love gnomes""}"

You can explicitly set the HTTP command using the -X command line option. For example, when creating a database, you set the name of the database in the URL you send using a PUT request:

shell> curl -X PUT http://user:pass@127.0.0.1:5984/demo
{"ok":true}

But to obtain the database information you use a GET request (with the return information formatted for clarity):

shell> curl -X GET http://user:pass@127.0.0.1:5984/demo
{
    "compact_running" : false,
    "doc_count" : 0,
    "db_name" : "demo",
    "purge_seq" : 0,
    "committed_update_seq" : 0,
    "doc_del_count" : 0,
    "disk_format_version" : 5,
    "update_seq" : 0,
    "instance_start_time" : "0",
    "disk_size" : 79
}

For certain operations, you must specify the content type of request, which you do by specifying the Content-Type header using the -H command-line option:

shell> curl -H 'Content-Type: application/json' http://127.0.0.1:5984/_uuids

You can also submit ‘payload’ data, that is, data in the body of the HTTP request using the -d option. This is useful if you need to submit JSON structures, for example document data, as part of the request. For example, to submit a simple document to the demo database:

shell> curl -H 'Content-Type: application/json' \
            -X POST http://user:pass@127.0.0.1:5984/demo \
            -d '{"company": "Example, Inc."}'
{"ok":true,"id":"8843faaf0b831d364278331bc3001bd8",
 "rev":"1-33b9fbce46930280dab37d672bbc8bb9"}

In the above example, the argument after the -d option is the JSON of the document we want to submit.

The document can be accessed by using the automatically generated document ID that was returned:

shell> curl -X GET http://user:pass@127.0.0.1:5984/demo/8843faaf0b831d364278331bc3001bd8
{"_id":"8843faaf0b831d364278331bc3001bd8",
 "_rev":"1-33b9fbce46930280dab37d672bbc8bb9",
 "company":"Example, Inc."}

The API samples in the api/basics show the HTTP command, URL and any payload information that needs to be submitted (and the expected return value). All of these examples can be reproduced using curl with the command-line examples shown above.

Security

Authentication

CouchDB also defines a set of requests that only admin users are allowed to do. If you have defined one or more specific admin users, CouchDB will ask for identification for certain requests:

Creating a New Admin User

WARNING:

To the end of your etc/local.ini file, after the [admins] line, add the text admin = password, so it looks like this:

[admins]
admin = password

(Don’t worry about the password being in plain text; we’ll come back to this.)

Now, restart CouchDB using the method appropriate for your operating system. You should now be able to access CouchDB using your new administrator account:

> curl http://admin:password@127.0.0.1:5984/_up
{"status":"ok","seeds":{}}

Great!

Let’s create an admin user through the HTTP API. We’ll call her anna, and her password is secret. Note the double quotes in the following code; they are needed to denote a string value for the configuration API:

> HOST="http://admin:password@127.0.0.1:5984"
> NODENAME="_local"
> curl -X PUT $HOST/_node/$NODENAME/_config/admins/anna -d '"secret"'
""

As per the _config API’s behavior, we’re getting the previous value for the config item we just wrote. Since our admin user didn’t exist, we get an empty string.

Please note that _local serves as an alias for the local node name, so for all configuration URLs, NODENAME may be set to _local, to interact with the local node’s configuration.

SEE ALSO:

Hashing Passwords

[admins]
admin = -pbkdf2-71c01cb429088ac1a1e95f3482202622dc1e53fe,226701bece4ae0fc9a373a5e02bf5d07,10
anna = -pbkdf2-2d86831c82b440b8887169bd2eebb356821d621b,5e11b9a9228414ab92541beeeacbf125,10

The hash is that big, ugly, long string that starts out with -pbkdf2-.

To compare a plain-text password during authentication with the stored hash, the hashing algorithm is run and the resulting hash is compared to the stored hash. The probability of two identical hashes for different passwords is too insignificant to mention (c.f. Bruce Schneier). Should the stored hash fall into the hands of an attacker, it is, by current standards, way too inconvenient (i.e., it’d take a lot of money and time) to find the plain-text password from the hash.

When CouchDB starts up, it reads a set of .ini files with config settings. It loads these settings into an internal data store (not a database). The config API lets you read the current configuration as well as change it and create new entries. CouchDB writes any changes back to the .ini files.

The .ini files can also be edited by hand when CouchDB is not running. Instead of creating the admin user as we showed previously, you could have stopped CouchDB, opened your local.ini, added anna = secret to the admins, and restarted CouchDB. Upon reading the new line from local.ini, CouchDB would run the hashing algorithm and write back the hash to local.ini, replacing the plain-text password — just as it did for our original admin user. To make sure CouchDB only hashes plain-text passwords and not an existing hash a second time, it prefixes the hash with -pbkdf2-, to distinguish between plain-text passwords and PBKDF2 hashed passwords. This means your plain-text password can’t start with the characters -pbkdf2-, but that’s pretty unlikely to begin with.

Basic Authentication

> HOST="http://127.0.0.1:5984"
> curl -X PUT $HOST/somedatabase
{"error":"unauthorized","reason":"You are not a server admin."}

That looks about right. Now we try again with the correct credentials:

> HOST="http://anna:secret@127.0.0.1:5984"
> curl -X PUT $HOST/somedatabase
{"ok":true}

If you have ever accessed a website or FTP server that was password-protected, the username:password@ URL variant should look familiar.

If you are security conscious, the missing s in http:// will make you nervous. We’re sending our password to CouchDB in plain text. This is a bad thing, right? Yes, but consider our scenario: CouchDB listens on 127.0.0.1 on a development box that we’re the sole user of. Who could possibly sniff our password?

If you are in a production environment, however, you need to reconsider. Will your CouchDB instance communicate over a public network? Even a LAN shared with other collocation customers is public. There are multiple ways to secure communication between you or your application and CouchDB that exceed the scope of this documentation. CouchDB as of version 1.1.0 comes with SSL built in.

SEE ALSO:

Cookie Authentication

To remedy some of these concerns, CouchDB supports cookie authentication. With cookie authentication your application doesn’t have to include the ugly login dialog that the users’ browsers come with. You can use a regular HTML form to submit logins to CouchDB. Upon receipt, CouchDB will generate a one-time token that the client can use in its next request to CouchDB. When CouchDB sees the token in a subsequent request, it will authenticate the user based on the token without the need to see the password again. By default, a token is valid for 10 minutes.

To obtain the first token and thus authenticate a user for the first time, the username and password must be sent to the _session API. The API is smart enough to decode HTML form submissions, so you don’t have to resort to any smarts in your application.

If you are not using HTML forms to log in, you need to send an HTTP request that looks as if an HTML form generated it. Luckily, this is super simple:

> HOST="http://127.0.0.1:5984"
> curl -vX POST $HOST/_session \
       -H 'Content-Type:application/x-www-form-urlencoded' \
       -d 'name=anna&password=secret'

CouchDB replies, and we’ll give you some more detail:

< HTTP/1.1 200 OK
< Set-Cookie: AuthSession=YW5uYTo0QUIzOTdFQjrC4ipN-D-53hw1sJepVzcVxnriEw;
< Version=1; Path=/; HttpOnly
> ...
<
{"ok":true}

A 200 OK response code tells us all is well, a Set-Cookie header includes the token we can use for the next request, and the standard JSON response tells us again that the request was successful.

Now we can use this token to make another request as the same user without sending the username and password again:

> curl -vX PUT $HOST/mydatabase \
       --cookie AuthSession=YW5uYTo0QUIzOTdFQjrC4ipN-D-53hw1sJepVzcVxnriEw \
       -H "X-CouchDB-WWW-Authenticate: Cookie" \
       -H "Content-Type:application/x-www-form-urlencoded"
{"ok":true}

You can keep using this token for 10 minutes by default. After 10 minutes you need to authenticate your user again. The token lifetime can be configured with the timeout (in seconds) setting in the chttpd_auth configuration section.

SEE ALSO:

Authentication Database

This special database is a system database. This means that while it shares the common database API, there are some special security-related constraints applied. Below is a list of how the authentication database is different from the other databases.

NOTE:

These draconian rules are necessary since CouchDB cares about its users’ personal information and will not disclose it to just anyone. Often, user documents contain system information like login, password hash and roles, apart from sensitive personal information like real name, email, phone, special internal identifications and more. This is not information that you want to share with the World.

Users Documents

Additionally, you may specify any custom fields that relate to the target user.

Why the org.couchdb.user: prefix?

For current CouchDB releases, all users belong to the same org.couchdb.user namespace and this cannot be changed. This may be changed in future releases.

Creating a New User

curl -X PUT http://localhost:5984/_users/org.couchdb.user:jan \
     -H "Accept: application/json" \
     -H "Content-Type: application/json" \
     -d '{"name": "jan", "password": "apple", "roles": [], "type": "user"}'

This curl command will produce the following HTTP request:

PUT /_users/org.couchdb.user:jan HTTP/1.1
Accept: application/json
Content-Length: 62
Content-Type: application/json
Host: localhost:5984
User-Agent: curl/7.31.0

And CouchDB responds with:

HTTP/1.1 201 Created
Cache-Control: must-revalidate
Content-Length: 83
Content-Type: application/json
Date: Fri, 27 Sep 2013 07:33:28 GMT
ETag: "1-e0ebfb84005b920488fc7a8cc5470cc0"
Location: http://localhost:5984/_users/org.couchdb.user:jan
Server: CouchDB (Erlang OTP)
{"ok":true,"id":"org.couchdb.user:jan","rev":"1-e0ebfb84005b920488fc7a8cc5470cc0"}

The document was successfully created! The user jan should now exist in our database. Let’s check if this is true:

curl -X POST http://localhost:5984/_session -d 'name=jan&password=apple'

CouchDB should respond with:

{"ok":true,"name":"jan","roles":[]}

This means that the username was recognized and the password’s hash matches with the stored one. If we specify an incorrect login and/or password, CouchDB will notify us with the following error message:

{"error":"unauthorized","reason":"Name or password is incorrect."}

Password Changing

Summarizing the above process - we need to get the document’s content, add the password field with the new password in plain text and then store the JSON result to the authentication database.

curl -X GET http://localhost:5984/_users/org.couchdb.user:jan

{
    "_id": "org.couchdb.user:jan",
    "_rev": "1-e0ebfb84005b920488fc7a8cc5470cc0",
    "derived_key": "e579375db0e0c6a6fc79cd9e36a36859f71575c3",
    "iterations": 10,
    "name": "jan",
    "password_scheme": "pbkdf2",
    "roles": [],
    "salt": "1112283cf988a34f124200a050d308a1",
    "type": "user"
}

Here is our user’s document. We may strip hashes from the stored document to reduce the amount of posted data:

curl -X PUT http://localhost:5984/_users/org.couchdb.user:jan \
     -H "Accept: application/json" \
     -H "Content-Type: application/json" \
     -H "If-Match: 1-e0ebfb84005b920488fc7a8cc5470cc0" \
     -d '{"name":"jan", "roles":[], "type":"user", "password":"orange"}'

{"ok":true,"id":"org.couchdb.user:jan","rev":"2-ed293d3a0ae09f0c624f10538ef33c6f"}

Updated! Now let’s check that the password was really changed:

curl -X POST http://localhost:5984/_session -d 'name=jan&password=apple'

CouchDB should respond with:

{"error":"unauthorized","reason":"Name or password is incorrect."}

Looks like the password apple is wrong, what about orange?

curl -X POST http://localhost:5984/_session -d 'name=jan&password=orange'

CouchDB should respond with:

{"ok":true,"name":"jan","roles":[]}

Hooray! You may wonder why this was so complex - we need to retrieve user’s document, add a special field to it, and post it back.

NOTE:

Authorization

Database authorization rules assign a user into one of two classes:

Note that a database admin is not the same as a server admin – the actions of a database admin are restricted to a specific database.

When a database is first created, there are no members or admins. HTTP requests that have no authentication credentials or have credentials for a normal user are treated as members, and those with server admin credentials are treated as database admins. To change the default permissions, you must create a _security document in the database:

> curl -X PUT http://localhost:5984/mydatabase/_security \
     -u anna:secret \
     -H "Content-Type: application/json" \
     -d '{"admins": { "names": [], "roles": [] }, "members": { "names": ["jan"], "roles": [] } }'

The HTTP request to create the _security document must contain the credentials of a server admin. CouchDB will respond with:

{"ok":true}

The database is now secured against anonymous reads and writes:

> curl http://localhost:5984/mydatabase/

{"error":"unauthorized","reason":"You are not authorized to access this db."}

You declared user “jan” as a member in this database, so he is able to read and write normal documents:

> curl -u jan:apple http://localhost:5984/mydatabase/

{"db_name":"mydatabase","doc_count":1,"doc_del_count":0,"update_seq":3,"purge_seq":0,
"compact_running":false,"sizes":{"active":272,"disk":12376,"external":350},
"instance_start_time":"0","disk_format_version":6,"committed_update_seq":3}

If Jan attempted to create a design doc, however, CouchDB would return a 401 Unauthorized error because the username “jan” is not in the list of admin names and the /_users/org.couchdb.user:jan document doesn’t contain a role that matches any of the declared admin roles. If you want to promote Jan to an admin, you can update the security document to add “jan” to the names array under admin. Keeping track of individual database admin usernames is tedious, though, so you would likely prefer to create a database admin role and assign that role to the org.couchdb.user:jan user document:

> curl -X PUT http://localhost:5984/mydatabase/_security \
     -u anna:secret \
     -H "Content-Type: application/json" \
     -d '{"admins": { "names": [], "roles": ["mydatabase_admin"] }, "members": { "names": [], "roles": [] } }'

See the _security document reference page for additional details about specifying database members and admins.

Getting Started

All Systems Are Go!

Make sure CouchDB is still running, and then do:

curl http://127.0.0.1:5984/

This issues a GET request to your newly installed CouchDB instance.

The reply should look something like:

{
  "couchdb": "Welcome",
  "version": "3.0.0",
  "git_sha": "83bdcf693",
  "uuid": "56f16e7c93ff4a2dc20eb6acc7000b71",
  "features": [
    "access-ready",
    "partitioned",
    "pluggable-storage-engines",
    "reshard",
    "scheduler"
  ],
  "vendor": {
    "name": "The Apache Software Foundation"
  }
}

Not all that spectacular. CouchDB is saying “hello” with the running version number.

Next, we can get a list of databases:

curl -X GET http://admin:password@127.0.0.1:5984/_all_dbs

All we added to the previous request is the _all_dbs string, and our admin user name and password (set when installing CouchDB).

The response should look like:

["_replicator","_users"]

NOTE:

Oh, that’s right, we didn’t create any user databases yet!

NOTE:

Let’s create a database:

curl -X PUT http://admin:password@127.0.0.1:5984/baseball

CouchDB will reply with:

{"ok":true}

Retrieving the list of databases again shows some useful results this time:

curl -X GET http://admin:password@127.0.0.1:5984/_all_dbs

["baseball"]

NOTE:

Let’s create another database:

curl -X PUT http://admin:password@127.0.0.1:5984/baseball

CouchDB will reply with:

{"error":"file_exists","reason":"The database could not be created,
the file already exists."}

We already have a database with that name, so CouchDB will respond with an error. Let’s try again with a different database name:

curl -X PUT http://admin:password@127.0.0.1:5984/plankton

CouchDB will reply with:

{"ok":true}

Retrieving the list of databases yet again shows some useful results:

curl -X GET http://admin:password@127.0.0.1:5984/_all_dbs

CouchDB will respond with:

["baseball", "plankton"]

To round things off, let’s delete the second database:

curl -X DELETE http://admin:password@127.0.0.1:5984/plankton

CouchDB will reply with:

{"ok":true}

The list of databases is now the same as it was before:

curl -X GET http://admin:password@127.0.0.1:5984/_all_dbs

CouchDB will respond with:

["baseball"]

For brevity, we’ll skip working with documents, as the next section covers a different and potentially easier way of working with CouchDB that should provide experience with this. As we work through the example, keep in mind that “under the hood” everything is being done by the application exactly as you have been doing here manually. Everything is done using GET, PUT, POST, and DELETE with a URI.

Welcome to Fauxton

To load Fauxton in your browser, visit:

http://127.0.0.1:5984/_utils/

and log in when prompted with your admin password.

In later documents, we’ll focus on using CouchDB from server-side languages such as Ruby and Python. As such, this document is a great opportunity to showcase an example of natively serving up a dynamic web application using nothing more than CouchDB’s integrated web server, something you may wish to do with your own applications.

The first thing we should do with a fresh installation of CouchDB is run the test suite to verify that everything is working properly. This assures us that any problems we may run into aren’t due to bothersome issues with our setup. By the same token, failures in the Fauxton test suite are a red flag, telling us to double-check our installation before attempting to use a potentially broken database server, saving us the confusion when nothing seems to be working quite like we expect!

To validate your installation, click on the Verify link on the left-hand side, then press the green Verify Installation button. All tests should pass with a check mark. If any fail, re-check your installation steps.

Your First Database and Document

After your database has been created, Fauxton will display a list of all its documents. This list will start out empty, so let’s create our first document. Click the plus sign next to “All Documents” and select the “New Doc” link. CouchDB will generate a UUID for you.

For demoing purposes, having CouchDB assign a UUID is fine. When you write your first programs, we recommend assigning your own UUIDs. If you rely on the server to generate the UUID and you end up making two POST requests because the first POST request bombed out, you might generate two docs and never find out about the first one because only the second one will be reported back. Generating your own UUIDs makes sure that you’ll never end up with duplicate documents.

Fauxton will display the newly created document, with its _id field. To create a new field, simply use the editor to write valid JSON. Add a new field by appending a comma to the _id value, then adding the text:

"hello": "my new value"

Click the green Create Document button to finalize creating the document.

You can experiment with other JSON values; e.g., [1, 2, "c"] or {"foo": "bar"}.

You’ll notice that the document’s _rev has been added. We’ll go into more detail about this in later documents, but for now, the important thing to note is that _rev acts like a safety feature when saving a document. As long as you and CouchDB agree on the most recent _rev of a document, you can successfully save your changes.

For clarity, you may want to display the contents of the document in the all document view. To enable this, from the upper-right corner of the window, select Options, then check the Include Docs option. Finally, press the Run Query button. The full document should be displayed along with the _id and _rev values.

Running a Mango Query

The index specifies which fields we want to be able to query on, and the selector includes the actual query parameters that define what we are looking for exactly.

Indexes are stored as rows that are kept sorted by the fields you specify. This makes retrieving data from a range of keys efficient even when there are thousands or millions of rows.

Before we can run an example query, we’ll need some data to run it on. We’ll create documents with information about movies. Let’s create documents for three movies. (Allow CouchDB to generate the _id and _rev fields.) Use Fauxton to create documents that have a final JSON structure that look like this:

{
    "_id": "00a271787f89c0ef2e10e88a0c0001f4",
    "type": "movie",
    "title": "My Neighbour Totoro",
    "year": 1988,
    "director": "miyazaki",
    "rating": 8.2
}

{
    "_id": "00a271787f89c0ef2e10e88a0c0003f0",
    "type": "movie",
    "title": "Kikis Delivery Service",
    "year": 1989,
    "director": "miyazaki",
    "rating": 7.8
}

{
    "_id": "00a271787f89c0ef2e10e88a0c00048b",
    "type": "movie",
    "title": "Princess Mononoke",
    "year": 1997,
    "director": "miyazaki",
    "rating": 8.4
}

Now we want to be able to find a movie by its release year, we need to create a Mango Index. To do this, go to “Run A Query with Mango” in the Database overview. Then click on “manage indexes”, and change the index field on the left to look like this:

{
   "index": {
      "fields": [
         "year"
      ]
   },
   "name": "year-json-index",
   "type": "json"
}

This defines an index on the field year and allows us to send queries for documents from a specific year.

Next, click on “edit query” and change the Mango Query to look like this:

{
   "selector": {
      "year": {
         "$eq": 1988
      }
   }
}

Then click on ”Run Query”.

The result should be a single result, the movie “My Neighbour Totoro” which has the year value of 1988. $eq here stands for “equal”.

NOTE:

You can also query for all movies during the 1980s, with this selector:

{
   "selector": {
      "year": {
         "$lt": 1990,
         "$gte": 1980
      }
   }
}

The result are the two movies from 1988 and 1989. $lt here means “lower than”, and $gte means “greater than or equal to”. The latter currently doesn’t have any effect, given that all of our movies are more recent than 1980, but this makes the query future-proof and allows us to add older movies later.

Triggering Replication

First we’ll need to create an empty database to be the target of replication. Return to the Databases overview and create a database called hello-replication. Now click “Replication” in the sidebar and choose hello-world as the source and hello-replication as the target. Click “Replicate” to replicate your database.

To view the result of your replication, click on the Databases tab again. You should see the hello-replication database has the same number of documents as the hello-world database, and it should take up roughly the same size as well.

NOTE:

Wrapping Up

But before we get there, we’ll have another look at CouchDB’s HTTP API – now with a magnifying glass. Let’s curl up on the couch and relax.

The Core API

We start out by revisiting the basic operations we ran in the previous document intro/tour, looking behind the scenes. We also show what Fauxton needs to do behind its user interface to give us the nice features we saw earlier.

This document is both an introduction to the core CouchDB API as well as a reference. If you can’t remember how to run a particular request or why some parameters are needed, you can always come back here and look things up (we are probably the heaviest users of this document).

While explaining the API bits and pieces, we sometimes need to take a larger detour to explain the reasoning for a particular request. This is a good opportunity for us to tell you why CouchDB works the way it does.

The API can be subdivided into the following sections. We’ll explore them individually:

Server

curl http://127.0.0.1:5984/

CouchDB replies, all excited to get going:

{
  "couchdb": "Welcome",
  "version": "3.0.0",
  "git_sha": "83bdcf693",
  "uuid": "56f16e7c93ff4a2dc20eb6acc7000b71",
  "features": [
    "access-ready",
    "partitioned",
    "pluggable-storage-engines",
    "reshard",
    "scheduler"
  ],
  "vendor": {
    "name": "The Apache Software Foundation"
  }
}

You get back a JSON string, that, if parsed into a native object or data structure of your programming language, gives you access to the welcome string and version information.

This is not terribly useful, but it illustrates nicely the way CouchDB behaves. You send an HTTP request and you receive a JSON string in the HTTP response as a result.

Databases

Now let’s make one! We want to store our favorite music albums, and we creatively give our database the name albums. Note that we’re now using the -X option again to tell curl to send a PUT request instead of the default GET request:

curl -X PUT http://admin:password@127.0.0.1:5984/albums

CouchDB replies:

{"ok":true}

That’s it. You created a database and CouchDB told you that all went well. What happens if you try to create a database that already exists? Let’s try to create that database again:

curl -X PUT http://admin:password@127.0.0.1:5984/albums

CouchDB replies:

{"error":"file_exists","reason":"The database could not be created, the file already exists."}

We get back an error. This is pretty convenient. We also learn a little bit about how CouchDB works. CouchDB stores each database in a single file. Very simple.

Let’s create another database, this time with curl’s -v (for “verbose”) option. The verbose option tells curl to show us not only the essentials – the HTTP response body – but all the underlying request and response details:

curl -vX PUT http://admin:password@127.0.0.1:5984/albums-backup

curl elaborates:

* About to connect() to 127.0.0.1 port 5984 (#0)
*   Trying 127.0.0.1... connected
* Connected to 127.0.0.1 (127.0.0.1) port 5984 (#0)
> PUT /albums-backup HTTP/1.1
> User-Agent: curl/7.16.3 (powerpc-apple-darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3
> Host: 127.0.0.1:5984
> Accept: */*
>
< HTTP/1.1 201 Created
< Server: CouchDB (Erlang/OTP)
< Date: Sun, 05 Jul 2009 22:48:28 GMT
< Content-Type: text/plain;charset=utf-8
< Content-Length: 12
< Cache-Control: must-revalidate
<
{"ok":true}
* Connection #0 to host 127.0.0.1 left intact
* Closing connection #0

What a mouthful. Let’s step through this line by line to understand what’s going on and find out what’s important. Once you’ve seen this output a few times, you’ll be able to spot the important bits more easily.

* About to connect() to 127.0.0.1 port 5984 (#0)

This is curl telling us that it is going to establish a TCP connection to the CouchDB server we specified in our request URI. Not at all important, except when debugging networking issues.

*   Trying 127.0.0.1... connected
* Connected to 127.0.0.1 (127.0.0.1) port 5984 (#0)

curl tells us it successfully connected to CouchDB. Again, not important if you aren’t trying to find problems with your network.

The following lines are prefixed with > and < characters. The > means the line was sent to CouchDB verbatim (without the actual >). The < means the line was sent back to curl by CouchDB.

> PUT /albums-backup HTTP/1.1

This initiates an HTTP request. Its method is PUT, the URI is /albums-backup, and the HTTP version is HTTP/1.1. There is also HTTP/1.0, which is simpler in some cases, but for all practical reasons you should be using HTTP/1.1.

Next, we see a number of request headers. These are used to provide additional details about the request to CouchDB.

> User-Agent: curl/7.16.3 (powerpc-apple-darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3

The User-Agent header tells CouchDB which piece of client software is doing the HTTP request. We don’t learn anything new: it’s curl. This header is often useful in web development when there are known errors in client implementations that a server might want to prepare the response for. It also helps to determine which platform a user is on. This information can be used for technical and statistical reasons. For CouchDB, the User-Agent header is irrelevant.

> Host: 127.0.0.1:5984

The Host header is required by HTTP 1.1. It tells the server the hostname that came with the request.

> Accept: */*

The Accept header tells CouchDB that curl accepts any media type. We’ll look into why this is useful a little later.

>

An empty line denotes that the request headers are now finished and the rest of the request contains data we’re sending to the server. In this case, we’re not sending any data, so the rest of the curl output is dedicated to the HTTP response.

< HTTP/1.1 201 Created

The first line of CouchDB’s HTTP response includes the HTTP version information (again, to acknowledge that the requested version could be processed), an HTTP status code, and a status code message. Different requests trigger different response codes. There’s a whole range of them telling the client (curl in our case) what effect the request had on the server. Or, if an error occurred, what kind of error. RFC 2616 (the HTTP 1.1 specification) defines clear behavior for response codes. CouchDB fully follows the RFC.

The 201 Created status code tells the client that the resource the request was made against was successfully created. No surprise here, but if you remember that we got an error message when we tried to create this database twice, you now know that this response could include a different response code. Acting upon responses based on response codes is a common practice. For example, all response codes of 400 Bad Request or larger tell you that some error occurred. If you want to shortcut your logic and immediately deal with the error, you could just check a >= 400 response code.

< Server: CouchDB (Erlang/OTP)

The Server header is good for diagnostics. It tells us which CouchDB version and which underlying Erlang version we are talking to. In general, you can ignore this header, but it is good to know it’s there if you need it.

< Date: Sun, 05 Jul 2009 22:48:28 GMT

The Date header tells you the time of the server. Since client and server time are not necessarily synchronized, this header is purely informational. You shouldn’t build any critical application logic on top of this!

< Content-Type: text/plain;charset=utf-8

The Content-Type header tells you which MIME type the HTTP response body is and its encoding. We already know CouchDB returns JSON strings. The appropriate Content-Type header is application/json. Why do we see text/plain? This is where pragmatism wins over purity. Sending an application/json Content-Type header will make a browser offer you the returned JSON for download instead of just displaying it. Since it is extremely useful to be able to test CouchDB from a browser, CouchDB sends a text/plain content type, so all browsers will display the JSON as text.

NOTE:

Do you remember the Accept request header and how it is set to */* to express interest in any MIME type? If you send Accept: application/json in your request, CouchDB knows that you can deal with a pure JSON response with the proper Content-Type header and will use it instead of text/plain.

< Content-Length: 12

The Content-Length header simply tells us how many bytes the response body has.

< Cache-Control: must-revalidate

This Cache-Control header tells you, or any proxy server between CouchDB and you, not to cache this response.

<

This empty line tells us we’re done with the response headers and what follows now is the response body.

{"ok":true}

We’ve seen this before.

* Connection #0 to host 127.0.0.1 left intact
* Closing connection #0

The last two lines are curl telling us that it kept the TCP connection it opened in the beginning open for a moment, but then closed it after it received the entire response.

Throughout the documents, we’ll show more requests with the -v option, but we’ll omit some of the headers we’ve seen here and include only those that are important for the particular request.

Creating databases is all fine, but how do we get rid of one? Easy – just change the HTTP method:

> curl -vX DELETE http://admin:password@127.0.0.1:5984/albums-backup

This deletes a CouchDB database. The request will remove the file that the database contents are stored in. There is no “Are you sure?” safety net or any “Empty the trash” magic you’ve got to do to delete a database. Use this command with care. Your data will be deleted without a chance to bring it back easily if you don’t have a backup copy.

This section went knee-deep into HTTP and set the stage for discussing the rest of the core CouchDB API. Next stop: documents.

Documents

Each document in CouchDB has an ID. This ID is unique per database. You are free to choose any string to be the ID, but for best results we recommend a UUID (or GUID), i.e., a Universally (or Globally) Unique IDentifier. UUIDs are random numbers that have such a low collision probability that everybody can make thousands of UUIDs a minute for millions of years without ever creating a duplicate. This is a great way to ensure two independent people cannot create two different documents with the same ID. Why should you care what somebody else is doing? For one, that somebody else could be you at a later time or on a different computer; secondly, CouchDB replication lets you share documents with others and using UUIDs ensures that it all works. But more on that later; let’s make some documents:

curl -X PUT http://admin:password@127.0.0.1:5984/albums/6e1295ed6c29495e54cc05947f18c8af -d '{"title":"There is Nothing Left to Lose","artist":"Foo Fighters"}'

CouchDB replies:

{"ok":true,"id":"6e1295ed6c29495e54cc05947f18c8af","rev":"1-2902191555"}

The curl command appears complex, but let’s break it down. First, -X PUT tells curl to make a PUT request. It is followed by the URL that specifies your CouchDB IP address and port. The resource part of the URL /albums/6e1295ed6c29495e54cc05947f18c8af specifies the location of a document inside our albums database. The wild collection of numbers and characters is a UUID. This UUID is your document’s ID. Finally, the -d flag tells curl to use the following string as the body for the PUT request. The string is a simple JSON structure including title and artist attributes with their respective values.

NOTE:

curl -X GET http://127.0.0.1:5984/_uuids

{"uuids":["6e1295ed6c29495e54cc05947f18c8af"]}

To double-check that CouchDB isn’t lying about having saved your document (it usually doesn’t), try to retrieve it by sending a GET request:

curl -X GET http://admin:password@127.0.0.1:5984/albums/6e1295ed6c29495e54cc05947f18c8af

We hope you see a pattern here. Everything in CouchDB has an address, a URI, and you use the different HTTP methods to operate on these URIs.

CouchDB replies:

{"_id":"6e1295ed6c29495e54cc05947f18c8af","_rev":"1-2902191555","title":"There is Nothing Left to Lose","artist":"Foo Fighters"}

This looks a lot like the document you asked CouchDB to save, which is good. But you should notice that CouchDB added two fields to your JSON structure. The first is _id, which holds the UUID we asked CouchDB to save our document under. We always know the ID of a document if it is included, which is very convenient.

The second field is _rev. It stands for revision.

Revisions

If you want to update or delete a document, CouchDB expects you to include the _rev field of the revision you wish to change. When CouchDB accepts the change, it will generate a new revision number. This mechanism ensures that, in case somebody else made a change without you knowing before you got to request the document update, CouchDB will not accept your update because you are likely to overwrite data you didn’t know existed. Or simplified: whoever saves a change to a document first, wins. Let’s see what happens if we don’t provide a _rev field (which is equivalent to providing a outdated value):

curl -X PUT http://admin:password@127.0.0.1:5984/albums/6e1295ed6c29495e54cc05947f18c8af \
     -d '{"title":"There is Nothing Left to Lose","artist":"Foo Fighters","year":"1997"}'

CouchDB replies:

{"error":"conflict","reason":"Document update conflict."}

If you see this, add the latest revision number of your document to the JSON structure:

curl -X PUT http://admin:password@127.0.0.1:5984/albums/6e1295ed6c29495e54cc05947f18c8af \
     -d '{"_rev":"1-2902191555","title":"There is Nothing Left to Lose","artist":"Foo Fighters","year":"1997"}'

Now you see why it was handy that CouchDB returned that _rev when we made the initial request. CouchDB replies:

{"ok":true,"id":"6e1295ed6c29495e54cc05947f18c8af","rev":"2-8aff9ee9d06671fa89c99d20a4b3ae"}

CouchDB accepted your write and also generated a new revision number. The revision number is the MD5 hash of the transport representation of a document with an N- prefix denoting the number of times a document got updated. This is useful for replication. See replication/conflicts for more information.

There are multiple reasons why CouchDB uses this revision system, which is also called Multi-Version Concurrency Control (MVCC). They all work hand-in-hand, and this is a good opportunity to explain some of them.

One of the aspects of the HTTP protocol that CouchDB uses is that it is stateless. What does that mean? When talking to CouchDB you need to make requests. Making a request includes opening a network connection to CouchDB, exchanging bytes, and closing the connection. This is done every time you make a request. Other protocols allow you to open a connection, exchange bytes, keep the connection open, exchange more bytes later – maybe depending on the bytes you exchanged at the beginning – and eventually close the connection. Holding a connection open for later use requires the server to do extra work. One common pattern is that for the lifetime of a connection, the client has a consistent and static view of the data on the server. Managing huge amounts of parallel connections is a significant amount of work. HTTP connections are usually short-lived, and making the same guarantees is a lot easier. As a result, CouchDB can handle many more concurrent connections.

Another reason CouchDB uses MVCC is that this model is simpler conceptually and, as a consequence, easier to program. CouchDB uses less code to make this work, and less code is always good because the ratio of defects per lines of code is static.

The revision system also has positive effects on replication and storage mechanisms, but we’ll explore these later in the documents.

WARNING:

Documents in Detail

We’ll add some more of our favorite music albums. Get a fresh UUID from the /_uuids resource. If you don’t remember how that works, you can look it up a few pages back.

curl -vX PUT http://admin:password@127.0.0.1:5984/albums/70b50bfa0a4b3aed1f8aff9e92dc16a0 \
     -d '{"title":"Blackened Sky","artist":"Biffy Clyro","year":2002}'

NOTE:

Now with the -v option, CouchDB’s reply (with only the important bits shown) looks like this:

> PUT /albums/70b50bfa0a4b3aed1f8aff9e92dc16a0 HTTP/1.1
>
< HTTP/1.1 201 Created
< Location: http://127.0.0.1:5984/albums/70b50bfa0a4b3aed1f8aff9e92dc16a0
< ETag: "1-e89c99d29d06671fa0a4b3ae8aff9e"
<
{"ok":true,"id":"70b50bfa0a4b3aed1f8aff9e92dc16a0","rev":"1-e89c99d29d06671fa0a4b3ae8aff9e"}

We’re getting back the 201 Created HTTP status code in the response headers, as we saw earlier when we created a database. The Location header gives us a full URL to our newly created document. And there’s a new header. An ETag in HTTP-speak identifies a specific version of a resource. In this case, it identifies a specific version (the first one) of our new document. Sound familiar? Yes, conceptually, an ETag is the same as a CouchDB document revision number, and it shouldn’t come as a surprise that CouchDB uses revision numbers for ETags. ETags are useful for caching infrastructures.

Attachments

Attachments get their own URL where you can upload data. Say we want to add the album artwork to the 6e1295ed6c29495e54cc05947f18c8af document (“There is Nothing Left to Lose”), and let’s also say the artwork is in a file artwork.jpg in the current directory:

curl -vX PUT http://admin:password@127.0.0.1:5984/albums/6e1295ed6c29495e54cc05947f18c8af/artwork.jpg?rev=2-2739352689 \
     --data-binary @artwork.jpg -H "Content-Type:image/jpg"

NOTE:

You should now see your artwork image if you point your browser to http://127.0.0.1:5984/albums/6e1295ed6c29495e54cc05947f18c8af/artwork.jpg

If you request the document again, you’ll see a new member:

curl http://admin:password@127.0.0.1:5984/albums/6e1295ed6c29495e54cc05947f18c8af

CouchDB replies:

{
    "_id": "6e1295ed6c29495e54cc05947f18c8af",
    "_rev": "3-131533518",
    "title": "There is Nothing Left to Lose",
    "artist": "Foo Fighters",
    "year": "1997",
    "_attachments": {
        "artwork.jpg": {
            "stub": true,
            "content_type": "image/jpg",
            "length": 52450
        }
    }
}

_attachments is a list of keys and values where the values are JSON objects containing the attachment metadata. stub=true tells us that this entry is just the metadata. If we use the ?attachments=true HTTP option when requesting this document, we’d get a Base64 encoded string containing the attachment data.

We’ll have a look at more document request options later as we explore more features of CouchDB, such as replication, which is the next topic.

Replication

In a simple POST request, you tell CouchDB the source and the target of a replication and CouchDB will figure out which documents and new document revisions are on source that are not yet on target, and will proceed to move the missing documents and revisions over.

We’ll take an in-depth look at replication in the document replication/intro; in this document, we’ll just show you how to use it.

First, we’ll create a target database. Note that CouchDB won’t automatically create a target database for you, and will return a replication failure if the target doesn’t exist (likewise for the source, but that mistake isn’t as easy to make):

curl -X PUT http://admin:password@127.0.0.1:5984/albums-replica

Now we can use the database albums-replica as a replication target:

curl -vX POST http://admin:password@127.0.0.1:5984/_replicate \
     -d '{"source":"http://127.0.0.1:5984/albums","target":"http://127.0.0.1:5984/albums-replica"}' \
     -H "Content-Type: application/json"

NOTE:

NOTE:

CouchDB replies (this time we formatted the output so you can read it more easily):

{
    "history": [
        {
            "start_last_seq": 0,
            "missing_found": 2,
            "docs_read": 2,
            "end_last_seq": 5,
            "missing_checked": 2,
            "docs_written": 2,
            "doc_write_failures": 0,
            "end_time": "Sat, 11 Jul 2009 17:36:21 GMT",
            "start_time": "Sat, 11 Jul 2009 17:36:20 GMT"
        }
    ],
    "source_last_seq": 5,
    "session_id": "924e75e914392343de89c99d29d06671",
    "ok": true
}

CouchDB maintains a session history of replications. The response for a replication request contains the history entry for this replication session. It is also worth noting that the request for replication will stay open until replication closes. If you have a lot of documents, it’ll take a while until they are all replicated and you won’t get back the replication response until all documents are replicated. It is important to note that replication replicates the database only as it was at the point in time when replication was started. So, any additions, modifications, or deletions subsequent to the start of replication will not be replicated.

We’ll punt on the details again – the "ok": true at the end tells us all went well. If you now have a look at the albums-replica database, you should see all the documents that you created in the albums database. Neat, eh?

What you just did is called local replication in CouchDB terms. You created a local copy of a database. This is useful for backups or to keep snapshots of a specific state of your data around for later. You might want to do this if you are developing your applications but want to be able to roll back to a stable version of your code and data.

There are more types of replication useful in other situations. The source and target members of our replication request are actually links (like in HTML) and so far we’ve seen links relative to the server we’re working on (hence local). You can also specify a remote database as the target:

curl -vX POST http://admin:password@127.0.0.1:5984/_replicate \
     -d '{"source":"http://127.0.0.1:5984/albums","target":"http://example.org:5984/albums-replica"}' \
     -H "Content-Type:application/json"

Using a local source and a remote target database is called push replication. We’re pushing changes to a remote server.

NOTE:

This is great for sharing local changes with remote servers or buddies next door.

You can also use a remote source and a local target to do a pull replication. This is great for getting the latest changes from a server that is used by others:

curl -vX POST http://admin:password@127.0.0.1:5984/_replicate \
     -d '{"source":"http://example.org:5984/albums-replica","target":"http://127.0.0.1:5984/albums"}' \
     -H "Content-Type:application/json"

Finally, you can run remote replication, which is mostly useful for management operations:

curl -vX POST http://admin:password@127.0.0.1:5984/_replicate \
     -d '{"source":"http://example.org:5984/albums","target":"http://example.org:5984/albums-replica"}' \
     -H"Content-Type: application/json"

NOTE:

Wrapping Up

SEE ALSO:

Introduction to Replication

Replication involves a source and a destination database, which can be on the same or on different CouchDB instances. The aim of replication is that at the end of the process, all active documents in the source database are also in the destination database and all documents that were deleted in the source database are also deleted in the destination database (if they even existed).

Transient and Persistent Replication

Triggering, Stopping and Monitoring Replications

If a replication is currently running its status can be inspected through the active tasks API (see api/server/active_tasks, replication-status and api/server/_scheduler/jobs).

For document based-replications, api/server/_scheduler/docs can be used to get a complete state summary. This API is preferred as it will show the state of the replication document before it becomes a replication job.

For transient replications there is no way to query their state when the job is finished.

A replication can be stopped by deleting the document, or by updating it with its cancel property set to true.

Replication Procedure

A replication task will finish once it reaches the end of the changes feed. If its continuous property is set to true, it will wait for new changes to appear until the task is canceled. Replication tasks also create checkpoint documents on the destination to ensure that a restarted task can continue from where it stopped, for example after it has crashed.

When a replication task is initiated on the sending node, it is called push replication, if it is initiated by the receiving node, it is called pull replication.

Master - Master replication

Controlling which Documents to Replicate

Local documents are never replicated (see api/local).

selectorobj can be included in a replication document (see replication-settings). A selector object contains a query expression that is used to test whether a document should be replicated.

filterfun can be used in a replication (see replication-settings). The replication task evaluates the filter function for each document in the changes feed. The document is only replicated if the filter returns true.

NOTE:

NOTE:

Migrating Data to Clients

Replicator Database

Changed in version 3.2.0: Fair share scheduling was introduced. Multiple _replicator databases get an equal chance (configurable) of running their jobs. Previously replication jobs were scheduled without any regard of their originating database.

The _replicator database works like any other in CouchDB, but documents added to it will trigger replications. Create (PUT or POST) a document to start replication. DELETE a replication document to cancel an ongoing replication.

These documents have exactly the same content as the JSON objects we used to POST to _replicate (fields source, target, create_target, create_target_params, continuous, doc_ids, filter, query_params, use_checkpoints, checkpoint_interval).

Replication documents can have a user defined _id (handy for finding a specific replication request later). Design Documents (and _local documents) added to the replicator database are ignored.

The default replicator database is _replicator. Additional replicator databases can be created. To be recognized as such by the system, their database names should end with /_replicator.

Basics

{
    "_id": "my_rep",
    "source": "http://myserver.com/foo",
    "target": {
        "url": "http://localhost:5984/bar",
        "auth": {
            "basic": {
                "username": "user",
                "password": "pass"
            }
        }
    },
    "create_target":  true,
    "continuous": true
}

In the couch log you’ll see 2 entries like these:

[notice] 2017-04-05T17:16:19.646716Z node1@127.0.0.1 <0.29432.0> -------- Replication `"a81a78e822837e66df423d54279c15fe+continuous+create_target"` is using:
    4 worker processes
    a worker batch size of 500
    20 HTTP connections
    a connection timeout of 30000 milliseconds
    10 retries per request
    socket options are: [{keepalive,true},{nodelay,false}]
[notice] 2017-04-05T17:16:19.646759Z node1@127.0.0.1 <0.29432.0> -------- Document `my_rep` triggered replication `a81a78e822837e66df423d54279c15fe+continuous+create_target`

Replication state of this document can then be queried from http://adm:pass@localhost:5984/_scheduler/docs/_replicator/my_rep

{
    "database": "_replicator",
    "doc_id": "my_rep",
    "error_count": 0,
    "id": "a81a78e822837e66df423d54279c15fe+continuous+create_target",
    "info": {
        "revisions_checked": 113,
        "missing_revisions_found": 113,
        "docs_read": 113,
        "docs_written": 113,
        "changes_pending": 0,
        "doc_write_failures": 0,
        "checkpointed_source_seq": "113-g1AAAACTeJzLYWBgYMpgTmHgz8tPSTV0MDQy1zMAQsMckEQiQ1L9____szKYE01ygQLsZsYGqcamiZjKcRqRxwIkGRqA1H-oSbZgk1KMLCzTDE0wdWUBAF6HJIQ",
        "source_seq": "113-g1AAAACTeJzLYWBgYMpgTmHgz8tPSTV0MDQy1zMAQsMckEQiQ1L9____szKYE01ygQLsZsYGqcamiZjKcRqRxwIkGRqA1H-oSbZgk1KMLCzTDE0wdWUBAF6HJIQ",
        "through_seq": "113-g1AAAACTeJzLYWBgYMpgTmHgz8tPSTV0MDQy1zMAQsMckEQiQ1L9____szKYE01ygQLsZsYGqcamiZjKcRqRxwIkGRqA1H-oSbZgk1KMLCzTDE0wdWUBAF6HJIQ"
    },
    "last_updated": "2017-04-05T19:18:15Z",
    "node": "node1@127.0.0.1",
    "source_proxy": null,
    "target_proxy": null,
    "source": "http://myserver.com/foo/",
    "start_time": "2017-04-05T19:18:15Z",
    "state": "running",
    "target": "http://localhost:5984/bar/"
}

The state is running. That means replicator has scheduled this replication job to run. Replication document contents stay the same. Previously, before version 2.1, it was updated with the triggered state.

The replication job will also appear in

http://adm:pass@localhost:5984/_scheduler/jobs

{
    "jobs": [
        {
            "database": "_replicator",
            "doc_id": "my_rep",
            "history": [
                {
                    "timestamp": "2017-04-05T19:18:15Z",
                    "type": "started"
                },
                {
                    "timestamp": "2017-04-05T19:18:15Z",
                    "type": "added"
                }
            ],
            "id": "a81a78e822837e66df423d54279c15fe+continuous+create_target",
            "info": {
                "changes_pending": 0,
                "checkpointed_source_seq": "113-g1AAAACTeJzLYWBgYMpgTmHgz8tPSTV0MDQy1zMAQsMckEQiQ1L9____szKYE01ygQLsZsYGqcamiZjKcRqRxwIkGRqA1H-oSbZgk1KMLCzTDE0wdWUBAF6HJIQ",
                "doc_write_failures": 0,
                "docs_read": 113,
                "docs_written": 113,
                "missing_revisions_found": 113,
                "revisions_checked": 113,
                "source_seq": "113-g1AAAACTeJzLYWBgYMpgTmHgz8tPSTV0MDQy1zMAQsMckEQiQ1L9____szKYE01ygQLsZsYGqcamiZjKcRqRxwIkGRqA1H-oSbZgk1KMLCzTDE0wdWUBAF6HJIQ",
                "through_seq": "113-g1AAAACTeJzLYWBgYMpgTmHgz8tPSTV0MDQy1zMAQsMckEQiQ1L9____szKYE01ygQLsZsYGqcamiZjKcRqRxwIkGRqA1H-oSbZgk1KMLCzTDE0wdWUBAF6HJIQ"
            },
            "node": "node1@127.0.0.1",
            "pid": "<0.1174.0>",
            "source": "http://myserver.com/foo/",
            "start_time": "2017-04-05T19:18:15Z",
            "target": "http://localhost:5984/bar/",
            "user": null
        }
    ],
    "offset": 0,
    "total_rows": 1
}

_scheduler/jobs shows more information, such as a detailed history of state changes. If a persistent replication has not yet started, has failed, or is completed, information about its state can only be found in _scheduler/docs. Keep in mind that some replication documents could be invalid and could not become a replication job. Others might be delayed because they are fetching data from a slow source database.

If there is an error, for example if the source database is missing, the replication job will crash and retry after a wait period. Each successive crash will result in a longer waiting period.

For example, POST-ing this document

{
    "_id": "my_rep_crashing",
    "source": "http://myserver.com/missing",
    "target": {
        "url": "http://localhost:5984/bar",
        "auth": {
            "basic": {
                "username": "user",
                "password": "pass"
            }
        }
    },
    "create_target":  true,
    "continuous": true
}

when source database is missing, will result in periodic starts and crashes with an increasingly larger interval. The history list from _scheduler/jobs for this replication would look something like this:

[
      {
          "reason": "db_not_found: could not open http://adm:*****@localhost:5984/missing/",
          "timestamp": "2017-04-05T20:55:10Z",
          "type": "crashed"
      },
      {
          "timestamp": "2017-04-05T20:55:10Z",
          "type": "started"
      },
      {
          "reason": "db_not_found: could not open http://adm:*****@localhost:5984/missing/",
          "timestamp": "2017-04-05T20:47:10Z",
          "type": "crashed"
      },
      {
          "timestamp": "2017-04-05T20:47:10Z",
          "type": "started"
      }
]

_scheduler/docs shows a shorter summary:

{
      "database": "_replicator",
      "doc_id": "my_rep_crashing",
      "error_count": 6,
      "id": "cb78391640ed34e9578e638d9bb00e44+create_target",
      "info": {
           "error": "db_not_found: could not open http://myserver.com/missing/"
      },
      "last_updated": "2017-04-05T20:55:10Z",
      "node": "node1@127.0.0.1",
      "source_proxy": null,
      "target_proxy": null,
      "source": "http://myserver.com/missing/",
      "start_time": "2017-04-05T20:38:34Z",
      "state": "crashing",
      "target": "http://localhost:5984/bar/"
}

Repeated crashes are described as a crashing state. -ing suffix implies this is a temporary state. User at any moment could create the missing database and then replication job could return back to the normal.

Documents describing the same replication

{
    "_id": "my_rep",
    "source": "http://myserver.com/foo",
    "target":  "http://user:pass@localhost:5984/bar",
    "create_target":  true,
    "continuous": true
}

and

{
    "_id": "my_rep_dup",
    "source": "http://myserver.com/foo",
    "target":  "http://user:pass@localhost:5984/bar",
    "create_target":  true,
    "continuous": true
}

Both describe exactly the same replication (only their _ids differ). In this case document my_rep triggers the replication, while my_rep_dup` will fail. Inspecting _scheduler/docs explains exactly why it failed:

{
    "database": "_replicator",
    "doc_id": "my_rep_dup",
    "error_count": 1,
    "id": null,
    "info": {
        "error": "Replication `a81a78e822837e66df423d54279c15fe+continuous+create_target` specified by document `my_rep_dup` already started, triggered by document `my_rep` from db `_replicator`"
    },
    "last_updated": "2017-04-05T21:41:51Z",
    "source": "http://myserver.com/foo/",
    "start_time": "2017-04-05T21:41:51Z",
    "state": "failed",
    "target": "http://user:****@localhost:5984/bar",
}

Notice the state for this replication is failed. Unlike crashing, failed state is terminal. As long as both documents are present the replicator will not retry to run my_rep_dup replication. Another reason could be malformed documents. For example if worker process count is specified as a string ("worker_processes": "a few") instead of an integer, failure will occur.

Replication Scheduler

When deciding which jobs to stop and which to start, the scheduler uses a round-robin algorithm to ensure fairness. Jobs which have been running the longest time will be stopped, and jobs which have been waiting the longest time will be started.

NOTE:

The behavior of the scheduler can configured via max_jobs, interval and max_churn options. See Replicator configuration section for additional information.

Replication states

States descriptions

Blue vs yellow partitions states into “healthy” and “unhealthy”, respectively. Unhealthy states indicate something has gone wrong and it might need user’s attention.

Rectangle vs oval separates “terminal” states from “non-terminal” ones. Terminal states are those which will not transition to other states any more. Informally, jobs in a terminal state will not be retried and don’t consume memory or CPU resources.

Normal vs Continuous Replications

WARNING:

However, normal replications will still be stopped and rescheduled if an operator reduces the value for the maximum number of replications. This is so that if an operator decides replications are overwhelming a node that it has the ability to recover. Any stopped replications will be resubmitted to the queue to be rescheduled.

Compatibility Mode

[replicator]
update_docs = true

In this mode replicator will continue to write state updates to the documents.

To effectively disable the scheduling behavior, which periodically stop and starts jobs, set max_jobs configuration setting to a large number. For example:

[replicator]
max_jobs = 9999999

See Replicator configuration section for other replicator configuration options.

Canceling replications

Server restart

Clustering

Every time there is a cluster membership change, that is when nodes are added or removed, as it happens in a rolling reboot, replicator application will notice the change, rescan all the document and running replication, and re-evaluate their cluster placement in light of the new set of live nodes. This mechanism also provides replication fail-over in case a node fails. Replication jobs started from replication documents (but not those started from _replicate HTTP endpoint) will automatically migrate one of the live nodes.

Additional Replicator Databases

{
    "_id": "rep_from_A",
    "source":  "http://aserver.com:5984/foo",
    "target": {
        "url": "http://localhost:5984/foo_a",
        "auth": {
            "basic": {
                "username": "user",
                "password": "pass"
            }
        }
    },
    "continuous":  true
}

{
    "_id": "rep_from_B",
    "source":  "http://bserver.com:5984/foo",
    "target": {
        "url": "http://localhost:5984/foo_b",
        "auth": {
            "basic": {
                "username": "user",
                "password": "pass"
            }
        }
    },
    "continuous":  true
}

Now without stopping and restarting CouchDB, add another replicator database. For example another/_replicator:

$ curl -X PUT http://user:pass@localhost:5984/another%2F_replicator/
{"ok":true}

NOTE:

Then add a replication document to the new replicator database:

{
    "_id": "rep_from_X",
    "source":  "http://xserver.com:5984/foo",
    "target":  "http://user:pass@localhost:5984/foo_x",
    "continuous":  true
}

From now on, there are three replications active in the system: two replications from A and B, and a new one from X.

Then remove the additional replicator database:

$ curl -X DELETE http://user:pass@localhost:5984/another%2F_replicator/
{"ok":true}

After this operation, replication pulling from server X will be stopped and the replications in the _replicator database (pulling from servers A and B) will continue.

Fair Share Job Scheduling

This is accomplished by assigning a number of “shares” to each _replicator database and then automatically adjusting the proportion of running jobs to match each database’s proportion of shares. By default, each _replicator database is assigned 100 shares. It is possible to alter the share assignments for each individual _replicator database in the [replicator.shares] configuration section.

The fair share behavior is perhaps easier described with a set of examples. Each example assumes the default of max_jobs = 500, and two replicator databases: _replicator and another/_replicator.

Example 1: If _replicator has 1000 jobs and another/_replicator has 10, the scheduler will run about 490 jobs from _replicator and 10 jobs from another/_replicator.

Example 2: If _replicator has 200 jobs and another/_replicator also has 200 jobs, all 400 jobs will get to run as the sum of all the jobs is less than the max_jobs limit.

Example 3: If both replicator databases have 1000 jobs each, the scheduler will run about 250 jobs from each database on average.

Example 4: If both replicator databases have 1000 jobs each, but _replicator was assigned 400 shares, then on average the scheduler would run about 400 jobs from _replicator and 100 jobs from _another/replicator.

The proportions described in the examples are approximate and might oscillate a bit, and also might take anywhere from tens of minutes to an hour to converge.

Replicating the replicator database

{
     "_id": "rep_from_A",
     "source":  "http://aserver.com:5984/foo",
     "target":  "http://user:pass@localhost:5984/foo_a",
     "continuous":  true
}

{
     "_id": "rep_from_B",
     "source":  "http://bserver.com:5984/foo",
     "target":  "http://user:pass@localhost:5984/foo_b",
     "continuous":  true
}

Now you would like to have the same pull replications going on in server D, that is, you would like to have server D pull replicating from servers A and B. You have two options:

Both alternatives accomplish exactly the same goal.

Delegations

For admins, the user_ctx property is optional, and if it’s missing it defaults to a user context with name null and an empty list of roles, which means design documents won’t be written to local targets. If writing design documents to local targets is desired, the role _admin must be present in the user context’s list of roles.

Also, for admins the user_ctx property can be used to trigger a replication on behalf of another user. This is the user context that will be passed to local target database document validation functions.

NOTE:

Example delegated replication document:

{
    "_id": "my_rep",
    "source":  "http://bserver.com:5984/foo",
    "target":  "http://user:pass@localhost:5984/bar",
    "continuous":  true,
    "user_ctx": {
        "name": "joe",
        "roles": ["erlanger", "researcher"]
    }
}

As stated before, the user_ctx property is optional for admins, while being mandatory for regular (non-admin) users. When the roles property of user_ctx is missing, it defaults to the empty list [].

Selector Objects

The selector specifies fields in the document, and provides an expression to evaluate with the field content or other data. If the expression resolves to true, the document is replicated.

The selector object must:

The syntax for a selector is the same as the selectorsyntax used for _find.

Using a selector is significantly more efficient than using a JavaScript filter function, and is the recommended option if filtering on document attributes only.

Specifying Usernames and Passwords

{
    "target": {
        "url": "http://someurl.com/mydb",
        "auth": {
            "basic": {
                "username": "$username",
                "password": "$password"
             }
        }
    },
    ...
}

{
    "target":  "http://user:pass@localhost:5984/bar"
    ...
}

{
    "target": {
        "url": "http://someurl.com/mydb",
            "headers": {
                "Authorization": "Basic dXNlcjpwYXNz"
            }
        },
    ...
}

When credentials are provided in multiple forms, they are selected in the following order:

First, the auth object is checked, and if credentials are defined there, they are used. If they are not, then URL userinfo is checked. If credentials are found there, then those credentials are used, otherwise basic auth header is used.

Replication and conflict model

So on the desktop the document has Bob’s new E-mail address and his old mobile number, and on the laptop it has his old E-mail address and his new mobile number.

The question is, what happens to these conflicting updated documents?

CouchDB replication

So, Alice creates v1 and sync it. She updates to v2a on one side and v2b on the other, and then replicates. What happens?

The answer is simple: both versions exist on both sides!

  DESKTOP                          LAPTOP
+---------+
| /db/bob |                                     INITIAL
|   v1    |                                     CREATION
+---------+
+---------+                      +---------+
| /db/bob |  ----------------->  | /db/bob |     PUSH
|   v1    |                      |   v1    |
+---------+                      +---------+
+---------+                      +---------+  INDEPENDENT
| /db/bob |                      | /db/bob |     LOCAL
|   v2a   |                      |   v2b   |     EDITS
+---------+                      +---------+
+---------+                      +---------+
| /db/bob |  ----------------->  | /db/bob |     PUSH
|   v2a   |                      |   v2a   |
+---------+                      |   v2b   |
                                 +---------+
+---------+                      +---------+
| /db/bob |  <-----------------  | /db/bob |     PULL
|   v2a   |                      |   v2a   |
|   v2b   |                      |   v2b   |
+---------+                      +---------+

After all, this is not a file system, so there’s no restriction that only one document can exist with the name /db/bob. These are just “conflicting” revisions under the same name.

Because the changes are always replicated, the data is safe. Both machines have identical copies of both documents, so failure of a hard drive on either side won’t lose any of the changes.

Another thing to notice is that peers do not have to be configured or tracked. You can do regular replications to peers, or you can do one-off, ad-hoc pushes or pulls. After the replication has taken place, there is no record kept of which peer any particular document or revision came from.

So the question now is: what happens when you try to read /db/bob? By default, CouchDB picks one arbitrary revision as the “winner”, using a deterministic algorithm so that the same choice will be made on all peers. The same happens with views: the deterministically-chosen winner is the only revision fed into your map function.

Let’s say that the winner is v2a. On the desktop, if Alice reads the document she’ll see v2a, which is what she saved there. But on the laptop, after replication, she’ll also see only v2a. It could look as if the changes she made there have been lost - but of course they have not, they have just been hidden away as a conflicting revision. But eventually she’ll need these changes merged into Bob’s business card, otherwise they will effectively have been lost.

Any sensible business-card application will, at minimum, have to present the conflicting versions to Alice and allow her to create a new version incorporating information from them all. Ideally it would merge the updates itself.

Conflict avoidance

So imagine two users on the same node are fetching Bob’s business card, updating it concurrently, and writing it back:

USER1    ----------->  GET /db/bob
         <-----------  {"_rev":"1-aaa", ...}
USER2    ----------->  GET /db/bob
         <-----------  {"_rev":"1-aaa", ...}
USER1    ----------->  PUT /db/bob?rev=1-aaa
         <-----------  {"_rev":"2-bbb", ...}
USER2    ----------->  PUT /db/bob?rev=1-aaa
         <-----------  409 Conflict  (not saved)

User2’s changes are rejected, so it’s up to the app to fetch /db/bob again, and either:

So when working in this mode, your application still has to be able to handle these conflicts and have a suitable retry strategy, but these conflicts never end up inside the database itself.

Revision tree

  ,--> r2a
r1 --> r2b
  `--> r2c

Each branch can then extend its history - for example if you read revision r2b and then PUT with ?rev=r2b then you will make a new revision along that particular branch.

  ,--> r2a -> r3a -> r4a
r1 --> r2b -> r3b
  `--> r2c -> r3c

Here, (r4a, r3b, r3c) are the set of conflicting revisions. The way you resolve a conflict is to delete the leaf nodes along the other branches. So when you combine (r4a+r3b+r3c) into a single merged document, you would replace r4a and delete r3b and r3c.

  ,--> r2a -> r3a -> r4a -> r5a
r1 --> r2b -> r3b -> (r4b deleted)
  `--> r2c -> r3c -> (r4c deleted)

Note that r4b and r4c still exist as leaf nodes in the history tree, but as deleted docs. You can retrieve them but they will be marked "_deleted":true.

When you compact a database, the bodies of all the non-leaf documents are discarded. However, the list of historical _revs is retained, for the benefit of later conflict resolution in case you meet any old replicas of the database at some time in future. There is “revision pruning” to stop this getting arbitrarily large.

Working with conflicting documents

{
    "_id":"test",
    "_rev":"2-b91bb807b4685080c6a651115ff558f5",
    "hello":"bar"
}

If you do GET /db/test?conflicts=true, and the document is in a conflict state, then you will get the winner plus a _conflicts member containing an array of the revs of the other, conflicting revision(s). You can then fetch them individually using subsequent GET /db/test?rev=xxxx operations:

{
    "_id":"test",
    "_rev":"2-b91bb807b4685080c6a651115ff558f5",
    "hello":"bar",
    "_conflicts":[
        "2-65db2a11b5172bf928e3bcf59f728970",
        "2-5bc3c6319edf62d4c624277fdd0ae191"
    ]
}

If you do GET /db/test?open_revs=all then you will get all the leaf nodes of the revision tree. This will give you all the current conflicts, but will also give you leaf nodes which have been deleted (i.e. parts of the conflict history which have since been resolved). You can remove these by filtering out documents with "_deleted":true:

[
    {"ok":{"_id":"test","_rev":"2-5bc3c6319edf62d4c624277fdd0ae191","hello":"foo"}},
    {"ok":{"_id":"test","_rev":"2-65db2a11b5172bf928e3bcf59f728970","hello":"baz"}},
    {"ok":{"_id":"test","_rev":"2-b91bb807b4685080c6a651115ff558f5","hello":"bar"}}
]

The "ok" tag is an artifact of open_revs, which also lets you list explicit revisions as a JSON array, e.g. open_revs=[rev1,rev2,rev3]. In this form, it would be possible to request a revision which is now missing, because the database has been compacted.

NOTE:

Once you have retrieved all the conflicting revisions, your application can then choose to display them all to the user. Or it could attempt to merge them, write back the merged version, and delete the conflicting versions - that is, to resolve the conflict permanently.

As described above, you need to update one revision and delete all the conflicting revisions explicitly. This can be done using a single POST to _bulk_docs, setting "_deleted":true on those revisions you wish to delete.

Multiple document API

Finding conflicted documents with Mango

CouchDB’s Mango system allows easy querying of documents with conflicts, returning the full body of each document as well.

Here’s how to use it to find all conflicts in a database:

$ curl -X POST http://127.0.0.1/dbname/_find \
    -d '{"selector": {"_conflicts": { "$exists": true}}, "conflicts": true}' \
    -Hcontent-type:application/json

{"docs": [
{"_id":"doc","_rev":"1-3975759ccff3842adf690a5c10caee42","a":2,"_conflicts":["1-23202479633c2b380f79507a776743d5"]}
],
"bookmark": "g1AAAABheJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYozA1kgKQ6YVA5QkBFMgKSVDHWNjI0MjEzMLc2MjZONkowtDNLMLU0NzBPNzc3MTYxTTLOysgCY2ReV"}

The bookmark value can be used to navigate through additional pages of results if necessary. Mango by default only returns 25 results per request.

If you expect to run this query often, be sure to create a Mango secondary index to speed the query:

$ curl -X POST http://127.0.0.1/dbname/_index \
    -d '{"index":{"fields": ["_conflicts"]}}' \
    -Hcontent-type:application/json

Of course, the selector can be enhanced to filter documents on additional keys in the document. Be sure to add those keys to your secondary index as well, or a full database scan will be triggered.

Finding conflicted documents using the _all_docs index

$ curl 'http://127.0.0.1:5984/conflict_test/_all_docs?include_docs=true&conflicts=true'

{
    "total_rows":1,
    "offset":0,
    "rows":[
        {
            "id":"test",
            "key":"test",
            "value":{"rev":"2-b91bb807b4685080c6a651115ff558f5"},
            "doc":{
                "_id":"test",
                "_rev":"2-b91bb807b4685080c6a651115ff558f5",
                "hello":"bar"
            }
        }
    ]
}

$ curl 'http://127.0.0.1:5984/conflict_test/test?conflicts=true'

{
    "_id":"test",
    "_rev":"2-b91bb807b4685080c6a651115ff558f5",
    "hello":"bar",
    "_conflicts":[
        "2-65db2a11b5172bf928e3bcf59f728970",
        "2-5bc3c6319edf62d4c624277fdd0ae191"
    ]
}

View map functions

function(doc) {
    if (doc._conflicts) {
        emit(null, [doc._rev].concat(doc._conflicts));
    }
}

which gives the following output:

{
    "total_rows":1,
    "offset":0,
    "rows":[
        {
            "id":"test",
            "key":null,
            "value":[
                "2-b91bb807b4685080c6a651115ff558f5",
                "2-65db2a11b5172bf928e3bcf59f728970",
                "2-5bc3c6319edf62d4c624277fdd0ae191"
            ]
        }
    ]
}

If you do this, you can have a separate “sweep” process which periodically scans your database, looks for documents which have conflicts, fetches the conflicting revisions, and resolves them.

Whilst this keeps the main application simple, the problem with this approach is that there will be a window between a conflict being introduced and it being resolved. From a user’s viewpoint, this may appear that the document they just saved successfully may suddenly lose their changes, only to be resurrected some time later. This may or may not be acceptable.

Also, it’s easy to forget to start the sweeper, or not to implement it properly, and this will introduce odd behaviour which will be hard to track down.

CouchDB’s “winning” revision algorithm may mean that information drops out of a view until a conflict has been resolved. Consider Bob’s business card again; suppose Alice has a view which emits mobile numbers, so that her telephony application can display the caller’s name based on caller ID. If there are conflicting documents with Bob’s old and new mobile numbers, and they happen to be resolved in favour of Bob’s old number, then the view won’t be able to recognise his new one. In this particular case, the application might have preferred to put information from both the conflicting documents into the view, but this currently isn’t possible.

Suggested algorithm to fetch a document with conflict resolution:

This could either be done on every read (in which case you could replace all calls to GET in your application with calls to a library which does the above), or as part of your sweeper code.

And here is an example of this in Ruby using the low-level RestClient:

require 'rubygems'
require 'rest_client'
require 'json'
DB="http://127.0.0.1:5984/conflict_test"
# Write multiple documents
def writem(docs)
    JSON.parse(RestClient.post("#{DB}/_bulk_docs", {
        "docs" => docs,
    }.to_json))
end
# Write one document, return the rev
def write1(doc, id=nil, rev=nil)
    doc['_id'] = id if id
    doc['_rev'] = rev if rev
    writem([doc]).first['rev']
end
# Read a document, return *all* revs
def read1(id)
    retries = 0
    loop do
        # FIXME: escape id
        res = [JSON.parse(RestClient.get("#{DB}/#{id}?conflicts=true"))]
        if revs = res.first.delete('_conflicts')
            begin
                revs.each do |rev|
                    res << JSON.parse(RestClient.get("#{DB}/#{id}?rev=#{rev}"))
                end
            rescue
                retries += 1
                raise if retries >= 5
                next
            end
        end
        return res
    end
end
# Create DB
RestClient.delete DB rescue nil
RestClient.put DB, {}.to_json
# Write a document
rev1 = write1({"hello"=>"xxx"},"test")
p read1("test")
# Make three conflicting versions
write1({"hello"=>"foo"},"test",rev1)
write1({"hello"=>"bar"},"test",rev1)
write1({"hello"=>"baz"},"test",rev1)
res = read1("test")
p res
# Now let's replace these three with one
res.first['hello'] = "foo+bar+baz"
res.each_with_index do |r,i|
    unless i == 0
        r.replace({'_id'=>r['_id'], '_rev'=>r['_rev'], '_deleted'=>true})
    end
end
writem(res)
p read1("test")

An application written this way never has to deal with a PUT 409, and is automatically multi-master capable.

You can see that it’s straightforward enough when you know what you’re doing. It’s just that CouchDB doesn’t currently provide a convenient HTTP API for “fetch all conflicting revisions”, nor “PUT to supersede these N revisions”, so you need to wrap these yourself. At the time of writing, there are no known client-side libraries which provide support for this.

Merging and revision history

Some merge strategies look at the changes made to an object, compared to its previous version. This is how Git’s merge function works.

For example, to merge Bob’s business card versions v2a and v2b, you could look at the differences between v1 and v2b, and then apply these changes to v2a as well.

With CouchDB, you can sometimes get hold of old revisions of a document. For example, if you fetch /db/bob?rev=v2b&revs_info=true you’ll get a list of the previous revision ids which ended up with revision v2b. Doing the same for v2a you can find their common ancestor revision. However if the database has been compacted, the content of that document revision will have been lost. revs_info will still show that v1 was an ancestor, but report it as “missing”:

BEFORE COMPACTION           AFTER COMPACTION
     ,-> v2a                     v2a
   v1
     `-> v2b                     v2b

So if you want to work with diffs, the recommended way is to store those diffs within the new revision itself. That is: when you replace v1 with v2a, include an extra field or attachment in v2a which says which fields were changed from v1 to v2a. This unfortunately does mean additional book-keeping for your application.

Comparison with other replicating data stores

Unison

When you run unison, changes propagate both ways. If a file has changed on one side but not the other, the new replaces the old. Unison maintains a local state file so that it knows whether a file has changed since the last successful replication.

In our example it has changed on both sides. Only one file called bob.vcf can exist within the file system. Unison solves the problem by simply ducking out: the user can choose to replace the remote version with the local version, or vice versa (both of which would lose data), but the default action is to leave both sides unchanged.

From Alice’s point of view, at least this is a simple solution. Whenever she’s on the desktop she’ll see the version she last edited on the desktop, and whenever she’s on the laptop she’ll see the version she last edited there.

But because no replication has actually taken place, the data is not protected. If her laptop hard drive dies, she’ll lose all her changes made on the laptop; ditto if her desktop hard drive dies.

It’s up to her to copy across one of the versions manually (under a different filename), merge the two, and then finally push the merged version to the other side.

Note also that the original file (version v1) has been lost at this point. So it’s not going to be known from inspection alone whether v2a or v2b has the most up-to-date E-mail address for Bob, or which version has the most up-to-date mobile number. Alice has to remember which one she entered last.

Git

If you are replicating with one or more peers, a separate branch is made for each of those peers. For example, you might have:

main               -- my local branch
remotes/foo/main   -- branch on peer 'foo'
remotes/bar/main   -- branch on peer 'bar'

In the regular workflow, replication is a “pull”, importing changes from a remote peer into the local repository. A “pull” does two things: first “fetch” the state of the peer into the remote tracking branch for that peer; and then attempt to “merge” those changes into the local branch.

Now let’s consider the business card. Alice has created a Git repo containing bob.vcf, and cloned it across to the other machine. The branches look like this, where AAAAAAAA is the SHA1 of the commit:

---------- desktop ----------           ---------- laptop ----------
main: AAAAAAAA                        main: AAAAAAAA
remotes/laptop/main: AAAAAAAA         remotes/desktop/main: AAAAAAAA

Now she makes a change on the desktop, and commits it into the desktop repo; then she makes a different change on the laptop, and commits it into the laptop repo:

---------- desktop ----------           ---------- laptop ----------
main: BBBBBBBB                        main: CCCCCCCC
remotes/laptop/main: AAAAAAAA         remotes/desktop/main: AAAAAAAA

Now on the desktop she does git pull laptop. First, the remote objects are copied across into the local repo and the remote tracking branch is updated:

---------- desktop ----------           ---------- laptop ----------
main: BBBBBBBB                        main: CCCCCCCC
remotes/laptop/main: CCCCCCCC         remotes/desktop/main: AAAAAAAA

NOTE:

Then Git will attempt to merge the changes in. Knowing that the parent commit to CCCCCCCC is AAAAAAAA, it takes a diff between AAAAAAAA and CCCCCCCC and tries to apply it to BBBBBBBB.

If this is successful, then you’ll get a new version with a merge commit:

---------- desktop ----------           ---------- laptop ----------
main: DDDDDDDD                        main: CCCCCCCC
remotes/laptop/main: CCCCCCCC         remotes/desktop/main: AAAAAAAA

Then Alice has to logon to the laptop and run git pull desktop. A similar process occurs. The remote tracking branch is updated:

---------- desktop ----------           ---------- laptop ----------
main: DDDDDDDD                        main: CCCCCCCC
remotes/laptop/main: CCCCCCCC         remotes/desktop/main: DDDDDDDD

Then a merge takes place. This is a special case: CCCCCCCC is one of the parent commits of DDDDDDDD, so the laptop can fast forward update from CCCCCCCC to DDDDDDDD directly without having to do any complex merging. This leaves the final state as:

---------- desktop ----------           ---------- laptop ----------
main: DDDDDDDD                        main: DDDDDDDD
remotes/laptop/main: CCCCCCCC         remotes/desktop/main: DDDDDDDD

Now this is all and good, but you may wonder how this is relevant when thinking about CouchDB.

First, note what happens in the case when the merge algorithm fails. The changes are still propagated from the remote repo into the local one, and are available in the remote tracking branch. So, unlike Unison, you know the data is protected. It’s just that the local working copy may fail to update, or may diverge from the remote version. It’s up to you to create and commit the combined version yourself, but you are guaranteed to have all the history you might need to do this.

Note that while it is possible to build new merge algorithms into Git, the standard ones are focused on line-based changes to source code. They don’t work well for XML or JSON if it’s presented without any line breaks.

The other interesting consideration is multiple peers. In this case you have multiple remote tracking branches, some of which may match your local branch, some of which may be behind you, and some of which may be ahead of you (i.e. contain changes that you haven’t yet merged):

main: AAAAAAAA
remotes/foo/main: BBBBBBBB
remotes/bar/main: CCCCCCCC
remotes/baz/main: AAAAAAAA

Note that each peer is explicitly tracked, and therefore has to be explicitly created. If a peer becomes stale or is no longer needed, it’s up to you to remove it from your configuration and delete the remote tracking branch. This is different from CouchDB, which doesn’t keep any peer state in the database.

Another difference between CouchDB and Git is that it maintains all history back to time zero - Git compaction keeps diffs between all those versions in order to reduce size, but CouchDB discards them. If you are constantly updating a document, the size of a Git repo would grow forever. It is possible (with some effort) to use “history rewriting” to make Git forget commits earlier than a particular one.

What is the CouchDB replication protocol? Is it like Git?

Key points

If you know Git, then you know how Couch replication works. Replicating is very similar to pushing or pulling with distributed source managers like Git.

CouchDB replication does not have its own protocol. A replicator simply connects to two DBs as a client, then reads from one and writes to the other. Push replication is reading the local data and updating the remote DB; pull replication is vice versa.

Everything flows from the data model

The replication algorithm is trivial, uninteresting. A trained monkey could design it. It’s simple because the cleverness is the data model, which has these useful characteristics:

Final notes

At least one sentence in this writeup (possibly this one) is complete BS.

CouchDB Replication Protocol

The CouchDB Replication Protocol is a protocol for synchronising JSON documents between 2 peers over HTTP/1.1 by using the public CouchDB REST API and is based on the Apache CouchDB MVCC Data model.

Preface

Language

Goals

The secondary goal is to provide enough detailed information about the protocol to make it easy to build tools on any language and platform that can synchronize data with CouchDB.

Definitions

Replication Protocol Algorithm

The reference implementation, written in Erlang, is provided by the couch_replicator module in Apache CouchDB.

It is RECOMMENDED that one follow this algorithm specification, use the same HTTP endpoints, and run requests with the same parameters to provide a completely compatible implementation. Custom Replicator implementations MAY use different HTTP API endpoints and request parameters depending on their local specifics and they MAY implement only part of the Replication Protocol to run only Push or Pull Replication. However, while such solutions could also run the Replication process, they loose compatibility with the CouchDB Replicator.

Verify Peers

+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
' Verify Peers:                                                             '
'                                                                           '
'                404 Not Found   +--------------------------------+         '
'       +----------------------- |     Check Source Existence     |         '
'       |                        +--------------------------------+         '
'       |                        |          HEAD /source          |         '
'       |                        +--------------------------------+         '
'       |                          |                                        '
'       |                          | 200 OK                                 '
'       |                          v                                        '
'       |                        +--------------------------------+         '
'       |                        |     Check Target Existence     | ----+   '
'       |                        +--------------------------------+     |   '
'       |                        |         HEAD /target           |     |   '
'       |                        +--------------------------------+     |   '
'       |                          |                                    |   '
'       |                          | 404 Not Found                      |   '
'       v                          v                                    |   '
'   +-------+    No              +--------------------------------+     |   '
'   | Abort | <----------------- |         Create Target?         |     |   '
'   +-------+                    +--------------------------------+     |   '
'       ^                          |                                    |   '
'       |                          | Yes                                |   '
'       |                          v                                    |   '
'       |        Failure         +--------------------------------+     |   '
'       +----------------------- |          Create Target         |     |   '
'                                +--------------------------------+     |   '
'                                |           PUT /target          |     |   '
'                                +--------------------------------+     |   '
'                                  |                                    |   '
'                                  | 201 Created                 200 OK |   '
'                                  |                                    |   '
+ - - - - - - - - - - - - - - - -  | - - - - - - - - - - - - - - - - -  | - +
                                   |                                    |
+ - - - - - - - - - - - - - - - -  | - - - - - - - - - - - - - - - - -  | - +
' Get Peers Information:           |                                    |   '
'                                  +------------------------------------+   '
'                                  |                                        '
'                                  v                                        '
'                                +--------------------------------+         '
'                                |     Get Source Information     |         '
'                                +--------------------------------+         '
'                                                                           '
+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +

The Replicator MUST ensure that both Source and Target exist by using HEAD /{db} requests.

Check Source Existence

HEAD /source HTTP/1.1
Host: localhost:5984
User-Agent: CouchDB

HTTP/1.1 200 OK
Cache-Control: must-revalidate
Content-Type: application/json
Date: Sat, 05 Oct 2013 08:50:39 GMT
Server: CouchDB (Erlang/OTP)

Check Target Existence

HEAD /target HTTP/1.1
Host: localhost:5984
User-Agent: CouchDB

HTTP/1.1 200 OK
Cache-Control: must-revalidate
Content-Type: application/json
Date: Sat, 05 Oct 2013 08:51:11 GMT
Server: CouchDB (Erlang/OTP)

Create Target?

PUT /target HTTP/1.1
Accept: application/json
Host: localhost:5984
User-Agent: CouchDB

HTTP/1.1 201 Created
Content-Length: 12
Content-Type: application/json
Date: Sat, 05 Oct 2013 08:58:41 GMT
Server: CouchDB (Erlang/OTP)
{
    "ok": true
}

However, the Replicator’s PUT request MAY NOT succeeded due to insufficient privileges (which are granted by the provided credential) and so receive a 401 Unauthorized or a 403 Forbidden error. Such errors SHOULD be expected and well handled:

HTTP/1.1 500 Internal Server Error
Cache-Control: must-revalidate
Content-Length: 108
Content-Type: application/json
Date: Fri, 09 May 2014 13:50:32 GMT
Server: CouchDB (Erlang OTP)
{
    "error": "unauthorized",
    "reason": "unauthorized to access or create database http://localhost:5984/target"
}

Abort

HTTP/1.1 500 Internal Server Error
Cache-Control: must-revalidate
Content-Length: 56
Content-Type: application/json
Date: Sat, 05 Oct 2013 08:55:29 GMT
Server: CouchDB (Erlang OTP)
{
    "error": "db_not_found",
    "reason": "could not open source"
}

Get Peers Information

+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -+
' Verify Peers:                                                    '
'                         +------------------------+               '
'                         | Check Target Existence |               '
'                         +------------------------+               '
'                                     |                            '
'                                     | 200 OK                     '
'                                     |                            '
+ - - - - - - - - - - - - - - - - - - | - - - - - - - - - - - - - -+
                                      |
+ - - - - - - - - - - - - - - - - - - | - - - - - - - - - - - - - -+
' Get Peers Information:              |                            '
'                                     v                            '
'                         +------------------------+               '
'                         | Get Source Information |               '
'                         +------------------------+               '
'                         |      GET /source       |               '
'                         +------------------------+               '
'                                     |                            '
'                                     | 200 OK                     '
'                                     v                            '
'                         +------------------------+               '
'                         | Get Target Information |               '
'                         +------------------------+               '
'                         |      GET /target       |               '
'                         +------------------------+               '
'                                     |                            '
'                                     | 200 OK                     '
'                                     |                            '
+ - - - - - - - - - - - - - - - - - - | - - - - - - - - - - - - - -+
                                      |
+ - - - - - - - - - - - - - - - - - - | - - - - - - - - - - - - - -+
' Find Common Ancestry:               |                            '
'                                     |                            '
'                                     v                            '
'                         +-------------------------+              '
'                         | Generate Replication ID |              '
'                         +-------------------------+              '
'                                                                  '
+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -+

The Replicator retrieves basic information both from Source and Target using GET /{db} requests. The GET response MUST contain JSON objects with the following mandatory fields:

Any other fields are optional. The information that the Replicator needs is the update_seq field: this value will be used to define a temporary (because Database data is subject to change) upper bound for changes feed listening and statistic calculating to show proper Replication progress.

Get Source Information

GET /source HTTP/1.1
Accept: application/json
Host: localhost:5984
User-Agent: CouchDB

HTTP/1.1 200 OK
Cache-Control: must-revalidate
Content-Length: 256
Content-Type: application/json
Date: Tue, 08 Oct 2013 07:53:08 GMT
Server: CouchDB (Erlang OTP)
{
    "committed_update_seq": 61772,
    "compact_running": false,
    "db_name": "source",
    "disk_format_version": 6,
    "doc_count": 41961,
    "doc_del_count": 3807,
    "instance_start_time": "0",
    "purge_seq": 0,
    "sizes": {
      "active": 70781613961,
      "disk": 79132913799,
      "external": 72345632950
    },
    "update_seq": 61772
}

Get Target Information

GET /target/ HTTP/1.1
Accept: application/json
Host: localhost:5984
User-Agent: CouchDB

HTTP/1.1 200 OK
Content-Length: 363
Content-Type: application/json
Date: Tue, 08 Oct 2013 12:37:01 GMT
Server: CouchDB (Erlang/OTP)
{
    "compact_running": false,
    "db_name": "target",
    "disk_format_version": 5,
    "doc_count": 1832,
    "doc_del_count": 1,
    "instance_start_time": "0",
    "purge_seq": 0,
    "sizes": {
      "active": 50829452,
      "disk": 77001455,
      "external": 60326450
    },
    "update_seq": "1841-g1AAAADveJzLYWBgYMlgTmGQT0lKzi9KdUhJMtbLSs1LLUst0k"
}

Find Common Ancestry

+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
' Get Peers Information:                                                    '
'                                                                           '
'                             +-------------------------------------------+ '
'                             |           Get Target Information          | '
'                             +-------------------------------------------+ '
'                               |                                           '
+ - - - - - - - - - - - - - - - | - - - - - - - - - - - - - - - - - - - - - +
                                |
+ - - - - - - - - - - - - - - - | - - - - - - - - - - - - - - - - - - - - - +
' Find Common Ancestry:         v                                           '
'                             +-------------------------------------------+ '
'                             |          Generate Replication ID          | '
'                             +-------------------------------------------+ '
'                               |                                           '
'                               |                                           '
'                               v                                           '
'                             +-------------------------------------------+ '
'                             |      Get Replication Log from Source      | '
'                             +-------------------------------------------+ '
'                             |     GET /source/_local/replication-id     | '
'                             +-------------------------------------------+ '
'                               |                                           '
'                               | 200 OK                                    '
'                               | 404 Not Found                             '
'                               v                                           '
'                             +-------------------------------------------+ '
'                             |      Get Replication Log from Target      | '
'                             +-------------------------------------------+ '
'                             |     GET /target/_local/replication-id     | '
'                             +-------------------------------------------+ '
'                               |                                           '
'                               | 200 OK                                    '
'                               | 404 Not Found                             '
'                               v                                           '
'                             +-------------------------------------------+ '
'                             |          Compare Replication Logs         | '
'                             +-------------------------------------------+ '
'                               |                                           '
'                               | Use latest common sequence as start point '
'                               |                                           '
+ - - - - - - - - - - - - - - - | - - - - - - - - - - - - - - - - - - - - - +
                                |
                                |
+ - - - - - - - - - - - - - - - | - - - - - - - - - - - - - - - - - - - - - +
' Locate Changed Documents:     |                                           '
'                               |                                           '
'                               v                                           '
'                             +-------------------------------------------+ '
'                             |        Listen Source Changes Feed         | '
'                             +-------------------------------------------+ '
'                                                                           '
+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +

Generate Replication ID

The Replication ID generation algorithm is implementation specific. Whatever algorithm is used it MUST uniquely identify the Replication process. CouchDB’s Replicator, for example, uses the following factors in generating a Replication ID:

NOTE:

Retrieve Replication Logs from Source and Target

GET /source/_local/b3e44b920ee2951cb2e123b63044427a HTTP/1.1
Accept: application/json
Host: localhost:5984
User-Agent: CouchDB

HTTP/1.1 200 OK
Cache-Control: must-revalidate
Content-Length: 1019
Content-Type: application/json
Date: Thu, 10 Oct 2013 06:18:56 GMT
ETag: "0-8"
Server: CouchDB (Erlang OTP)
{
    "_id": "_local/b3e44b920ee2951cb2e123b63044427a",
    "_rev": "0-8",
    "history": [
        {
            "doc_write_failures": 0,
            "docs_read": 2,
            "docs_written": 2,
            "end_last_seq": 5,
            "end_time": "Thu, 10 Oct 2013 05:56:38 GMT",
            "missing_checked": 2,
            "missing_found": 2,
            "recorded_seq": 5,
            "session_id": "d5a34cbbdafa70e0db5cb57d02a6b955",
            "start_last_seq": 3,
            "start_time": "Thu, 10 Oct 2013 05:56:38 GMT"
        },
        {
            "doc_write_failures": 0,
            "docs_read": 1,
            "docs_written": 1,
            "end_last_seq": 3,
            "end_time": "Thu, 10 Oct 2013 05:56:12 GMT",
            "missing_checked": 1,
            "missing_found": 1,
            "recorded_seq": 3,
            "session_id": "11a79cdae1719c362e9857cd1ddff09d",
            "start_last_seq": 2,
            "start_time": "Thu, 10 Oct 2013 05:56:12 GMT"
        },
        {
            "doc_write_failures": 0,
            "docs_read": 2,
            "docs_written": 2,
            "end_last_seq": 2,
            "end_time": "Thu, 10 Oct 2013 05:56:04 GMT",
            "missing_checked": 2,
            "missing_found": 2,
            "recorded_seq": 2,
            "session_id": "77cdf93cde05f15fcb710f320c37c155",
            "start_last_seq": 0,
            "start_time": "Thu, 10 Oct 2013 05:56:04 GMT"
        }
    ],
    "replication_id_version": 3,
    "session_id": "d5a34cbbdafa70e0db5cb57d02a6b955",
    "source_last_seq": 5
}

The Replication Log SHOULD contain the following fields:

This request MAY fall with a 404 Not Found response:

GET /source/_local/b6cef528f67aa1a8a014dd1144b10e09 HTTP/1.1
Accept: application/json
Host: localhost:5984
User-Agent: CouchDB

HTTP/1.1 404 Object Not Found
Cache-Control: must-revalidate
Content-Length: 41
Content-Type: application/json
Date: Tue, 08 Oct 2013 13:31:10 GMT
Server: CouchDB (Erlang OTP)
{
    "error": "not_found",
    "reason": "missing"
}

That’s OK. This means that there is no information about the current Replication so it must not have been run previously and as such the Replicator MUST run a Full Replication.

Compare Replication Logs

If Source and Target has no common ancestry, the Replicator MUST run Full Replication.

Locate Changed Documents

+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
' Find Common Ancestry:                                                     '
'                                                                           '
'             +------------------------------+                              '
'             |   Compare Replication Logs   |                              '
'             +------------------------------+                              '
'                                          |                                '
'                                          |                                '
+ - - - - - - - - - - - - - - - - - - - -  |  - - - - - - - - - - - - - - - +
                                           |
+ - - - - - - - - - - - - - - - - - - - -  |  - - - - - - - - - - - - - - - +
' Locate Changed Documents:                |                                '
'                                          |                                '
'                                          |                                '
'                                          v                                '
'            +-------------------------------+                              '
'   +------> |     Listen to Changes Feed    | -----+                       '
'   |        +-------------------------------+      |                       '
'   |        |     GET  /source/_changes     |      |                       '
'   |        |     POST /source/_changes     |      |                       '
'   |        +-------------------------------+      |                       '
'   |                                      |        |                       '
'   |                                      |        |                       '
'   |                There are new changes |        | No more changes       '
'   |                                      |        |                       '
'   |                                      v        v                       '
'   |        +-------------------------------+    +-----------------------+ '
'   |        |     Read Batch of Changes     |    | Replication Completed | '
'   |        +-------------------------------+    +-----------------------+ '
'   |                                      |                                '
'   | No                                   |                                '
'   |                                      v                                '
'   |        +-------------------------------+                              '
'   |        |  Compare Documents Revisions  |                              '
'   |        +-------------------------------+                              '
'   |        |    POST /target/_revs_diff    |                              '
'   |        +-------------------------------+                              '
'   |                                      |                                '
'   |                               200 OK |                                '
'   |                                      v                                '
'   |        +-------------------------------+                              '
'   +------- |     Any Differences Found?    |                              '
'            +-------------------------------+                              '
'                                          |                                '
'                                      Yes |                                '
'                                          |                                '
+ - - - - - - - - - - - - - - - - - - - -  |  - - - - - - - - - - - - - - - +
                                           |
+ - - - - - - - - - - - - - - - - - - - -  |  - - - - - - - - - - - - - - - +
' Replicate Changes:                       |                                '
'                                          v                                '
'            +-------------------------------+                              '
'            |  Fetch Next Changed Document  |                              '
'            +-------------------------------+                              '
'                                                                           '
+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +

Listen to Changes Feed

Additionally, the filter query parameter MAY be specified to enable a filter function on Source side. Other custom parameters MAY also be provided.

Read Batch of Changes

Note, that Changes Feed output format is different for a request with feed=normal and with feed=continuous query parameter.

Normal Feed:

GET /source/_changes?feed=normal&style=all_docs&heartbeat=10000 HTTP/1.1
Accept: application/json
Host: localhost:5984
User-Agent: CouchDB

HTTP/1.1 200 OK
Cache-Control: must-revalidate
Content-Type: application/json
Date: Fri, 09 May 2014 16:20:41 GMT
Server: CouchDB (Erlang OTP)
Transfer-Encoding: chunked
{"results":[
{"seq":14,"id":"f957f41e","changes":[{"rev":"3-46a3"}],"deleted":true}
{"seq":29,"id":"ddf339dd","changes":[{"rev":"10-304b"}]}
{"seq":37,"id":"d3cc62f5","changes":[{"rev":"2-eec2"}],"deleted":true}
{"seq":39,"id":"f13bd08b","changes":[{"rev":"1-b35d"}]}
{"seq":41,"id":"e0a99867","changes":[{"rev":"2-c1c6"}]}
{"seq":42,"id":"a75bdfc5","changes":[{"rev":"1-967a"}]}
{"seq":43,"id":"a5f467a0","changes":[{"rev":"1-5575"}]}
{"seq":45,"id":"470c3004","changes":[{"rev":"11-c292"}]}
{"seq":46,"id":"b1cb8508","changes":[{"rev":"10-ABC"}]}
{"seq":47,"id":"49ec0489","changes":[{"rev":"157-b01f"},{"rev":"123-6f7c"}]}
{"seq":49,"id":"dad10379","changes":[{"rev":"1-9346"},{"rev":"6-5b8a"}]}
{"seq":50,"id":"73464877","changes":[{"rev":"1-9f08"}]}
{"seq":51,"id":"7ae19302","changes":[{"rev":"1-57bf"}]}
{"seq":63,"id":"6a7a6c86","changes":[{"rev":"5-acf6"}],"deleted":true}
{"seq":64,"id":"dfb9850a","changes":[{"rev":"1-102f"}]}
{"seq":65,"id":"c532afa7","changes":[{"rev":"1-6491"}]}
{"seq":66,"id":"af8a9508","changes":[{"rev":"1-3db2"}]}
{"seq":67,"id":"caa3dded","changes":[{"rev":"1-6491"}]}
{"seq":68,"id":"79f3b4e9","changes":[{"rev":"1-102f"}]}
{"seq":69,"id":"1d89d16f","changes":[{"rev":"1-3db2"}]}
{"seq":71,"id":"abae7348","changes":[{"rev":"2-7051"}]}
{"seq":77,"id":"6c25534f","changes":[{"rev":"9-CDE"},{"rev":"3-00e7"},{"rev":"1-ABC"}]}
{"seq":78,"id":"SpaghettiWithMeatballs","changes":[{"rev":"22-5f95"}]}
],
"last_seq":78}

Continuous Feed:

GET /source/_changes?feed=continuous&style=all_docs&heartbeat=10000 HTTP/1.1
Accept: application/json
Host: localhost:5984
User-Agent: CouchDB

HTTP/1.1 200 OK
Cache-Control: must-revalidate
Content-Type: application/json
Date: Fri, 09 May 2014 16:22:22 GMT
Server: CouchDB (Erlang OTP)
Transfer-Encoding: chunked
{"seq":14,"id":"f957f41e","changes":[{"rev":"3-46a3"}],"deleted":true}
{"seq":29,"id":"ddf339dd","changes":[{"rev":"10-304b"}]}
{"seq":37,"id":"d3cc62f5","changes":[{"rev":"2-eec2"}],"deleted":true}
{"seq":39,"id":"f13bd08b","changes":[{"rev":"1-b35d"}]}
{"seq":41,"id":"e0a99867","changes":[{"rev":"2-c1c6"}]}
{"seq":42,"id":"a75bdfc5","changes":[{"rev":"1-967a"}]}
{"seq":43,"id":"a5f467a0","changes":[{"rev":"1-5575"}]}
{"seq":45,"id":"470c3004","changes":[{"rev":"11-c292"}]}
{"seq":46,"id":"b1cb8508","changes":[{"rev":"10-ABC"}]}
{"seq":47,"id":"49ec0489","changes":[{"rev":"157-b01f"},{"rev":"123-6f7c"}]}
{"seq":49,"id":"dad10379","changes":[{"rev":"1-9346"},{"rev":"6-5b8a"}]}
{"seq":50,"id":"73464877","changes":[{"rev":"1-9f08"}]}
{"seq":51,"id":"7ae19302","changes":[{"rev":"1-57bf"}]}
{"seq":63,"id":"6a7a6c86","changes":[{"rev":"5-acf6"}],"deleted":true}
{"seq":64,"id":"dfb9850a","changes":[{"rev":"1-102f"}]}
{"seq":65,"id":"c532afa7","changes":[{"rev":"1-6491"}]}
{"seq":66,"id":"af8a9508","changes":[{"rev":"1-3db2"}]}
{"seq":67,"id":"caa3dded","changes":[{"rev":"1-6491"}]}
{"seq":68,"id":"79f3b4e9","changes":[{"rev":"1-102f"}]}
{"seq":69,"id":"1d89d16f","changes":[{"rev":"1-3db2"}]}
{"seq":71,"id":"abae7348","changes":[{"rev":"2-7051"}]}
{"seq":75,"id":"SpaghettiWithMeatballs","changes":[{"rev":"21-5949"}]}
{"seq":77,"id":"6c255","changes":[{"rev":"9-CDE"},{"rev":"3-00e7"},{"rev":"1-ABC"}]}
{"seq":78,"id":"SpaghettiWithMeatballs","changes":[{"rev":"22-5f95"}]}

For both Changes Feed formats record-per-line style is preserved to simplify iterative fetching and decoding JSON objects with less memory footprint.

Calculate Revision Difference

POST /target/_revs_diff HTTP/1.1
Accept: application/json
Content-Length: 287
Content-Type: application/json
Host: localhost:5984
User-Agent: CouchDB
{
    "baz": [
        "2-7051cbe5c8faecd085a3fa619e6e6337"
    ],
    "foo": [
        "3-6a540f3d701ac518d3b9733d673c5484"
    ],
    "bar": [
        "1-d4e501ab47de6b2000fc8a02f84a0c77",
        "1-967a00dff5e02add41819138abb3284d"
    ]
}

HTTP/1.1 200 OK
Cache-Control: must-revalidate
Content-Length: 88
Content-Type: application/json
Date: Fri, 25 Oct 2013 14:44:41 GMT
Server: CouchDB (Erlang/OTP)
{
    "baz": {
        "missing": [
            "2-7051cbe5c8faecd085a3fa619e6e6337"
        ]
    },
    "bar": {
        "missing": [
            "1-d4e501ab47de6b2000fc8a02f84a0c77"
        ]
    }
}

In the response the Replicator receives a Document ID – Revisions mapping, but only for Revisions that do not exist in Target and are REQUIRED to be transferred from Source.

If all Revisions in the request match the current state of the Documents then the response will contain an empty JSON object:

POST /target/_revs_diff HTTP/1.1
Accept: application/json
Content-Length: 160
Content-Type: application/json
Host: localhost:5984
User-Agent: CouchDB
{
    "foo": [
        "3-6a540f3d701ac518d3b9733d673c5484"
    ],
    "bar": [
        "1-967a00dff5e02add41819138abb3284d"
    ]
}

HTTP/1.1 200 OK
Cache-Control: must-revalidate
Content-Length: 2
Content-Type: application/json
Date: Fri, 25 Oct 2013 14:45:00 GMT
Server: CouchDB (Erlang/OTP)
{}

Replication Completed

HTTP/1.1 200 OK
Cache-Control: must-revalidate
Content-Length: 414
Content-Type: application/json
Date: Fri, 09 May 2014 15:14:19 GMT
Server: CouchDB (Erlang OTP)
{
    "history": [
        {
            "doc_write_failures": 2,
            "docs_read": 2,
            "docs_written": 0,
            "end_last_seq": 2939,
            "end_time": "Fri, 09 May 2014 15:14:19 GMT",
            "missing_checked": 1835,
            "missing_found": 2,
            "recorded_seq": 2939,
            "session_id": "05918159f64842f1fe73e9e2157b2112",
            "start_last_seq": 0,
            "start_time": "Fri, 09 May 2014 15:14:18 GMT"
        }
    ],
    "ok": true,
    "replication_id_version": 3,
    "session_id": "05918159f64842f1fe73e9e2157b2112",
    "source_last_seq": 2939
}

Replicate Changes

+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
' Locate Changed Documents:                                                       '
'                                                                                 '
'               +-------------------------------------+                           '
'               |      Any Differences Found?         |                           '
'               +-------------------------------------+                           '
'                                                   |                             '
'                                                   |                             '
'                                                   |                             '
+ - - - - - - - - - - - - - - - - - - - - - - - - - | - - - - - - - - - - - - - - +
                                                    |
+ - - - - - - - - - - - - - - - - - - - - - - - - - | - - - - - - - - - - - - - - +
' Replicate Changes:                                |                             '
'                                                   v                             '
'               +-------------------------------------+                           '
'   +---------> |     Fetch Next Changed Document     | <---------------------+   '
'   |           +-------------------------------------+                       |   '
'   |           |          GET /source/docid          |                       |   '
'   |           +-------------------------------------+                       |   '
'   |             |                                                           |   '
'   |             |                                                           |   '
'   |             |                                          201 Created      |   '
'   |             | 200 OK                                   401 Unauthorized |   '
'   |             |                                          403 Forbidden    |   '
'   |             |                                                           |   '
'   |             v                                                           |   '
'   |           +-------------------------------------+                       |   '
'   |   +------ |  Document Has Changed Attachments?  |                       |   '
'   |   |       +-------------------------------------+                       |   '
'   |   |         |                                                           |   '
'   |   |         |                                                           |   '
'   |   |         | Yes                                                       |   '
'   |   |         |                                                           |   '
'   |   |         v                                                           |   '
'   |   |       +------------------------+   Yes    +---------------------------+ '
'   |   | No    |  Are They Big Enough?  | -------> | Update Document on Target | '
'   |   |       +------------------------+          +---------------------------+ '
'   |   |         |                                 |     PUT /target/docid     | '
'   |   |         |                                 +---------------------------+ '
'   |   |         |                                                               '
'   |   |         | No                                                            '
'   |   |         |                                                               '
'   |   |         v                                                               '
'   |   |       +-------------------------------------+                           '
'   |   +-----> |     Put Document Into the Stack     |                           '
'   |           +-------------------------------------+                           '
'   |             |                                                               '
'   |             |                                                               '
'   |             v                                                               '
'   |     No    +-------------------------------------+                           '
'   +---------- |           Stack is Full?            |                           '
'   |           +-------------------------------------+                           '
'   |             |                                                               '
'   |             | Yes                                                           '
'   |             |                                                               '
'   |             v                                                               '
'   |           +-------------------------------------+                           '
'   |           | Upload Stack of Documents to Target |                           '
'   |           +-------------------------------------+                           '
'   |           |       POST /target/_bulk_docs       |                           '
'   |           +-------------------------------------+                           '
'   |             |                                                               '
'   |             | 201 Created                                                   '
'   |             v                                                               '
'   |           +-------------------------------------+                           '
'   |           |          Ensure in Commit           |                           '
'   |           +-------------------------------------+                           '
'   |           |  POST /target/_ensure_full_commit   |                           '
'   |           +-------------------------------------+                           '
'   |             |                                                               '
'   |             | 201 Created                                                   '
'   |             v                                                               '
'   |           +-------------------------------------+                           '
'   |           |    Record Replication Checkpoint    |                           '
'   |           +-------------------------------------+                           '
'   |           |  PUT /source/_local/replication-id  |                           '
'   |           |  PUT /target/_local/replication-id  |                           '
'   |           +-------------------------------------+                           '
'   |             |                                                               '
'   |             | 201 Created                                                   '
'   |             v                                                               '
'   |     No    +-------------------------------------+                           '
'   +---------- | All Documents from Batch Processed? |                           '
'               +-------------------------------------+                           '
'                                                   |                             '
'                                               Yes |                             '
'                                                   |                             '
+ - - - - - - - - - - - - - - - - - - - - - - - - - | - - - - - - - - - - - - - - +
                                                    |
+ - - - - - - - - - - - - - - - - - - - - - - - - - | - - - - - - - - - - - - - - +
' Locate Changed Documents:                         |                             '
'                                                   v                             '
'               +-------------------------------------+                           '
'               |       Listen to Changes Feed        |                           '
'               +-------------------------------------+                           '
'                                                                                 '
+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +

Fetch Changed Documents

To fetch the Document the Replicator will make a GET /{db}/{docid} request with the following query parameters:

In the response Source SHOULD return multipart/mixed or respond instead with application/json unless the Accept header specifies a different mime type. The multipart/mixed content type allows handling the response data as a stream, since there could be multiple documents (one per each Leaf Revision) plus several attachments. These attachments are mostly binary and JSON has no way to handle such data except as base64 encoded strings which are very ineffective for transfer and processing operations.

With a multipart/mixed response the Replicator handles multiple Document Leaf Revisions and their attachments one by one as raw data without any additional encoding applied. There is also one agreement to make data processing more effective: the Document ALWAYS goes before its attachments, so the Replicator has no need to process all the data to map related Documents-Attachments and may handle it as stream with lesser memory footprint.

GET /source/SpaghettiWithMeatballs?revs=true&open_revs=[%225-00ecbbc%22,%221-917fa23%22,%223-6bcedf1%22]&latest=true HTTP/1.1
Accept: multipart/mixed
Host: localhost:5984
User-Agent: CouchDB

HTTP/1.1 200 OK
Content-Type: multipart/mixed; boundary="7b1596fc4940bc1be725ad67f11ec1c4"
Date: Thu, 07 Nov 2013 15:10:16 GMT
Server: CouchDB (Erlang OTP)
Transfer-Encoding: chunked
--7b1596fc4940bc1be725ad67f11ec1c4
Content-Type: application/json
{
    "_id": "SpaghettiWithMeatballs",
    "_rev": "1-917fa23",
    "_revisions": {
        "ids": [
            "917fa23"
        ],
        "start": 1
    },
    "description": "An Italian-American delicious dish",
    "ingredients": [
        "spaghetti",
        "tomato sauce",
        "meatballs"
    ],
    "name": "Spaghetti with meatballs"
}
--7b1596fc4940bc1be725ad67f11ec1c4
Content-Type: multipart/related; boundary="a81a77b0ca68389dda3243a43ca946f2"
--a81a77b0ca68389dda3243a43ca946f2
Content-Type: application/json
{
    "_attachments": {
      "recipe.txt": {
          "content_type": "text/plain",
          "digest": "md5-R5CrCb6fX10Y46AqtNn0oQ==",
          "follows": true,
          "length": 87,
          "revpos": 7
      }
    },
    "_id": "SpaghettiWithMeatballs",
    "_rev": "7-474f12e",
    "_revisions": {
        "ids": [
            "474f12e",
            "5949cfc",
            "00ecbbc",
            "fc997b6",
            "3552c87",
            "404838b",
            "5defd9d",
            "dc1e4be"
        ],
        "start": 7
    },
    "description": "An Italian-American delicious dish",
    "ingredients": [
        "spaghetti",
        "tomato sauce",
        "meatballs",
        "love"
    ],
    "name": "Spaghetti with meatballs"
}
--a81a77b0ca68389dda3243a43ca946f2
Content-Disposition: attachment; filename="recipe.txt"
Content-Type: text/plain
Content-Length: 87
1. Cook spaghetti
2. Cook meetballs
3. Mix them
4. Add tomato sauce
5. ...
6. PROFIT!
--a81a77b0ca68389dda3243a43ca946f2--
--7b1596fc4940bc1be725ad67f11ec1c4
Content-Type: application/json; error="true"
{"missing":"3-6bcedf1"}
--7b1596fc4940bc1be725ad67f11ec1c4--

After receiving the response, the Replicator puts all the received data into a local stack for further bulk upload to utilize network bandwidth effectively. The local stack size could be limited by number of Documents or bytes of handled JSON data. When the stack is full the Replicator uploads all the handled Document in bulk mode to the Target. While bulk operations are highly RECOMMENDED to be used, in certain cases the Replicator MAY upload Documents to Target one by one.

NOTE:

Upload Batch of Changed Documents

The request also MAY contain X-Couch-Full-Commit that used to control CouchDB <3.0 behavior when delayed commits were enabled. Other Peers MAY ignore this header or use it to control similar local feature.

POST /target/_bulk_docs HTTP/1.1
Accept: application/json
Content-Length: 826
Content-Type:application/json
Host: localhost:5984
User-Agent: CouchDB
X-Couch-Full-Commit: false
{
    "docs": [
        {
            "_id": "SpaghettiWithMeatballs",
            "_rev": "1-917fa2381192822767f010b95b45325b",
            "_revisions": {
                "ids": [
                    "917fa2381192822767f010b95b45325b"
                ],
                "start": 1
            },
            "description": "An Italian-American delicious dish",
            "ingredients": [
                "spaghetti",
                "tomato sauce",
                "meatballs"
            ],
            "name": "Spaghetti with meatballs"
        },
        {
            "_id": "LambStew",
            "_rev": "1-34c318924a8f327223eed702ddfdc66d",
            "_revisions": {
                "ids": [
                    "34c318924a8f327223eed702ddfdc66d"
                ],
                "start": 1
            },
            "servings": 6,
            "subtitle": "Delicious with scone topping",
            "title": "Lamb Stew"
        },
        {
            "_id": "FishStew",
            "_rev": "1-9c65296036141e575d32ba9c034dd3ee",
            "_revisions": {
                "ids": [
                    "9c65296036141e575d32ba9c034dd3ee"
                ],
                "start": 1
            },
            "servings": 4,
            "subtitle": "Delicious with fresh bread",
            "title": "Fish Stew"
        }
    ],
    "new_edits": false
}

In its response Target MUST return a JSON array with a list of Document update statuses. If the Document has been stored successfully, the list item MUST contain the field ok with true value. Otherwise it MUST contain error and reason fields with error type and a human-friendly reason description.

Document updating failure isn’t fatal as Target MAY reject the update for its own reasons. It’s RECOMMENDED to use error type forbidden for rejections, but other error types can also be used (like invalid field name etc.). The Replicator SHOULD NOT retry uploading rejected documents unless there are good reasons for doing so (e.g. there is special error type for that).

Note that while a update may fail for one Document in the response, Target can still return a 201 Created response. Same will be true if all updates fail for all uploaded Documents.

HTTP/1.1 201 Created
Cache-Control: must-revalidate
Content-Length: 246
Content-Type: application/json
Date: Sun, 10 Nov 2013 19:02:26 GMT
Server: CouchDB (Erlang/OTP)
[
    {
        "ok": true,
        "id": "SpaghettiWithMeatballs",
        "rev":" 1-917fa2381192822767f010b95b45325b"
    },
    {
        "ok": true,
        "id": "FishStew",
        "rev": "1-9c65296036141e575d32ba9c034dd3ee"
    },
    {
        "error": "forbidden",
        "id": "LambStew",
        "reason": "sorry",
        "rev": "1-34c318924a8f327223eed702ddfdc66d"
    }
]

Upload Document with Attachments

For this case the Replicator issues a /{db}/{docid}?new_edits=false request with multipart/related content type. Such a request allows one to easily stream the Document and all its attachments one by one without any serialization overhead.

PUT /target/SpaghettiWithMeatballs?new_edits=false HTTP/1.1
Accept: application/json
Content-Length: 1030
Content-Type: multipart/related; boundary="864d690aeb91f25d469dec6851fb57f2"
Host: localhost:5984
User-Agent: CouchDB
--2fa48cba80d0cdba7829931fe8acce9d
Content-Type: application/json
{
    "_attachments": {
        "recipe.txt": {
            "content_type": "text/plain",
            "digest": "md5-R5CrCb6fX10Y46AqtNn0oQ==",
            "follows": true,
            "length": 87,
            "revpos": 7
        }
    },
    "_id": "SpaghettiWithMeatballs",
    "_rev": "7-474f12eb068c717243487a9505f6123b",
    "_revisions": {
        "ids": [
            "474f12eb068c717243487a9505f6123b",
            "5949cfcd437e3ee22d2d98a26d1a83bf",
            "00ecbbc54e2a171156ec345b77dfdf59",
            "fc997b62794a6268f2636a4a176efcd6",
            "3552c87351aadc1e4bea2461a1e8113a",
            "404838bc2862ce76c6ebed046f9eb542",
            "5defd9d813628cea6e98196eb0ee8594"
        ],
        "start": 7
    },
    "description": "An Italian-American delicious dish",
    "ingredients": [
        "spaghetti",
        "tomato sauce",
        "meatballs",
        "love"
    ],
    "name": "Spaghetti with meatballs"
}
--2fa48cba80d0cdba7829931fe8acce9d
Content-Disposition: attachment; filename="recipe.txt"
Content-Type: text/plain
Content-Length: 87
1. Cook spaghetti
2. Cook meetballs
3. Mix them
4. Add tomato sauce
5. ...
6. PROFIT!
--2fa48cba80d0cdba7829931fe8acce9d--

HTTP/1.1 201 Created
Cache-Control: must-revalidate
Content-Length: 105
Content-Type: application/json
Date: Fri, 08 Nov 2013 16:35:27 GMT
Server: CouchDB (Erlang/OTP)
{
    "ok": true,
    "id": "SpaghettiWithMeatballs",
    "rev": "7-474f12eb068c717243487a9505f6123b"
}

Unlike bulk updating via POST /{db}/_bulk_docs endpoint, the response MAY come with a different status code. For instance, in the case when the Document is rejected, Target SHOULD respond with a 403 Forbidden:

HTTP/1.1 403 Forbidden
Cache-Control: must-revalidate
Content-Length: 39
Content-Type: application/json
Date: Fri, 08 Nov 2013 16:35:27 GMT
Server: CouchDB (Erlang/OTP)
{
    "error": "forbidden",
    "reason": "sorry"
}

Replicator SHOULD NOT retry requests in case of a 401 Unauthorized, 403 Forbidden, 409 Conflict or 412 Precondition Failed since repeating the request couldn’t solve the issue with user credentials or uploaded data.

Ensure In Commit

POST /target/_ensure_full_commit HTTP/1.1
Accept: application/json
Content-Type: application/json
Host: localhost:5984

HTTP/1.1 201 Created
Cache-Control: must-revalidate
Content-Length: 53
Content-Type: application/json
Date: Web, 06 Nov 2013 18:20:43 GMT
Server: CouchDB (Erlang/OTP)
{
    "instance_start_time": "0",
    "ok": true
}

Record Replication Checkpoint

Replicator updates Replication Log on Source:

PUT /source/_local/afa899a9e59589c3d4ce5668e3218aef HTTP/1.1
Accept: application/json
Content-Length: 591
Content-Type: application/json
Host: localhost:5984
User-Agent: CouchDB
{
    "_id": "_local/afa899a9e59589c3d4ce5668e3218aef",
    "_rev": "0-1",
    "_revisions": {
        "ids": [
            "31f36e40158e717fbe9842e227b389df"
        ],
        "start": 1
    },
    "history": [
        {
            "doc_write_failures": 0,
            "docs_read": 6,
            "docs_written": 6,
            "end_last_seq": 26,
            "end_time": "Thu, 07 Nov 2013 09:42:17 GMT",
            "missing_checked": 6,
            "missing_found": 6,
            "recorded_seq": 26,
            "session_id": "04bf15bf1d9fa8ac1abc67d0c3e04f07",
            "start_last_seq": 0,
            "start_time": "Thu, 07 Nov 2013 09:41:43 GMT"
        }
    ],
    "replication_id_version": 3,
    "session_id": "04bf15bf1d9fa8ac1abc67d0c3e04f07",
    "source_last_seq": 26
}

HTTP/1.1 201 Created
Cache-Control: must-revalidate
Content-Length: 75
Content-Type: application/json
Date: Thu, 07 Nov 2013 09:42:17 GMT
Server: CouchDB (Erlang/OTP)
{
    "id": "_local/afa899a9e59589c3d4ce5668e3218aef",
    "ok": true,
    "rev": "0-2"
}

…and on Target too:

PUT /target/_local/afa899a9e59589c3d4ce5668e3218aef HTTP/1.1
Accept: application/json
Content-Length: 591
Content-Type: application/json
Host: localhost:5984
User-Agent: CouchDB
{
    "_id": "_local/afa899a9e59589c3d4ce5668e3218aef",
    "_rev": "1-31f36e40158e717fbe9842e227b389df",
    "_revisions": {
        "ids": [
            "31f36e40158e717fbe9842e227b389df"
        ],
        "start": 1
    },
    "history": [
        {
            "doc_write_failures": 0,
            "docs_read": 6,
            "docs_written": 6,
            "end_last_seq": 26,
            "end_time": "Thu, 07 Nov 2013 09:42:17 GMT",
            "missing_checked": 6,
            "missing_found": 6,
            "recorded_seq": 26,
            "session_id": "04bf15bf1d9fa8ac1abc67d0c3e04f07",
            "start_last_seq": 0,
            "start_time": "Thu, 07 Nov 2013 09:41:43 GMT"
        }
    ],
    "replication_id_version": 3,
    "session_id": "04bf15bf1d9fa8ac1abc67d0c3e04f07",
    "source_last_seq": 26
}

HTTP/1.1 201 Created
Cache-Control: must-revalidate
Content-Length: 106
Content-Type: application/json
Date: Thu, 07 Nov 2013 09:42:17 GMT
Server: CouchDB (Erlang/OTP)
{
    "id": "_local/afa899a9e59589c3d4ce5668e3218aef",
    "ok": true,
    "rev": "2-9b5d1e36bed6ae08611466e30af1259a"
}

Continue Reading Changes

For Continuous Replication, the Replicator MUST continue to wait for new changes from Source.

Protocol Robustness

Error Responses

Bad Request

{
    "error": "bad_request",
    "reason": "invalid json"
}

Unauthorized

{
    "error": "unauthorized",
    "reason": "Name or password is incorrect"
}

Forbidden

{
    "error": "forbidden",
    "reason": "You may only update your own user document."
}

Resource Not Found

{
    "error": "not_found",
    "reason": "database \"target\" does not exists"
}

Method Not Allowed

{
    "error": "method_not_allowed",
    "reason": "Only GET, PUT, DELETE allowed"
}

Resource Conflict

{
    "error": "conflict",
    "reason": "document update conflict"
}

Precondition Failed

{
    "error": "db_exists",
    "reason": "database \"target\" exists"
}

Server Error

{
    "error": "worker_died",
    "reason": "kaboom!"
}

Optimisations

API Reference

Common Methods

For Target

For Source

Reference

Design Documents

But before we start to write our first document, let’s take a look at the list of common objects that will be used during our code journey - we’ll be using them extensively within each function:

Creation and Structure

Design documents are denoted by an id field with the format _design/{name}. Their structure follows the example below.

Example:

{
    "_id": "_design/example",
    "views": {
        "view-number-one": {
            "map": "function (doc) {/* function code here - see below */}"
        },
        "view-number-two": {
            "map": "function (doc) {/* function code here - see below */}",
            "reduce": "function (keys, values, rereduce) {/* function code here - see below */}"
        }
    },
    "updates": {
        "updatefun1": "function(doc,req) {/* function code here - see below */}",
        "updatefun2": "function(doc,req) {/* function code here - see below */}"
    },
    "filters": {
        "filterfunction1": "function(doc, req){ /* function code here - see below */ }"
    },
    "validate_doc_update": "function(newDoc, oldDoc, userCtx, secObj) { /* function code here - see below */ }",
    "language": "javascript"
}

As you can see, a design document can include multiple functions of the same type. The example defines two views, both of which have a map function and one of which has a reduce function. It also defines two update functions and one filter function. The Validate Document Update function is a special case, as each design document cannot contain more than one of those.

View Functions

Map Functions

Map functions accept a single document as the argument and (optionally) emit() key/value pairs that are stored in a view.

function (doc) {
  if (doc.type === 'post' && doc.tags && Array.isArray(doc.tags)) {
    doc.tags.forEach(function (tag) {
      emit(tag.toLowerCase(), 1);
    });
  }
}

In this example a key/value pair is emitted for each value in the tags array of a document with a type of “post”. Note that emit() may be called many times for a single document, so the same document may be available by several different keys.

Also keep in mind that each document is sealed to prevent the situation where one map function changes document state and another receives a modified version.

For efficiency reasons, documents are passed to a group of map functions - each document is processed by a group of map functions from all views of the related design document. This means that if you trigger an index update for one view in the design document, all others will get updated too.

Since version 1.1.0, map supports CommonJS modules and the require() function.

Reduce and Rereduce Functions

Reduce functions take two required arguments of keys and values lists - the result of the related map function - and an optional third value which indicates if rereduce mode is active or not. Rereduce is used for additional reduce values list, so when it is true there is no information about related keys (first argument is null).

Note that if the result of a reduce function is longer than the initial values list then a Query Server error will be raised. However, this behavior can be disabled by setting reduce_limit config option to false:

[query_server_config]
reduce_limit = false

While disabling reduce_limit might be useful for debug proposes, remember that the main task of reduce functions is to reduce the mapped result, not to make it bigger. Generally, your reduce function should converge rapidly to a single value - which could be an array or similar object.

Built-in Reduce Functions

New in version 2.2.

Aproximates the number of distinct keys in a view index using a variant of the HyperLogLog algorithm. This algorithm enables an efficient, parallelizable computation of cardinality using fixed memory resources. CouchDB has configured the underlying data structure to have a relative error of ~2%.

As this reducer ignores the emitted values entirely, an invocation with group=true will simply return a value of 1 for every distinct key in the view. In the case of array keys, querying the view with a group_level specified will return the number of distinct keys that share the common group prefix in each row. The algorithm is also cognizant of the startkey and endkey boundaries and will return the number of distinct keys within the specified key range.

A final note regarding Unicode collation: this reduce function uses the binary representation of each key in the index directly as input to the HyperLogLog filter. As such, it will (incorrectly) consider keys that are not byte identical but that compare equal according to the Unicode collation rules to be distinct keys, and thus has the potential to overestimate the cardinality of the key space if a large number of such keys exist.

Counts the number of values in the index with a given key. This could be implemented in JavaScript as:

// could be replaced by _count
function(keys, values, rereduce) {
    if (rereduce) {
        return sum(values);
    } else {
        return values.length;
    }
}

Computes the following quantities for numeric values associated with each key: sum, min, max, count, and sumsqr. The behavior of the _stats function varies depending on the output of the map function. The simplest case is when the map phase emits a single numeric value for each key. In this case the _stats function is equivalent to the following JavaScript:

// could be replaced by _stats
function(keys, values, rereduce) {
    if (rereduce) {
        return {
            'sum': values.reduce(function(a, b) { return a + b.sum }, 0),
            'min': values.reduce(function(a, b) { return Math.min(a, b.min) }, Infinity),
            'max': values.reduce(function(a, b) { return Math.max(a, b.max) }, -Infinity),
            'count': values.reduce(function(a, b) { return a + b.count }, 0),
            'sumsqr': values.reduce(function(a, b) { return a + b.sumsqr }, 0)
        }
    } else {
        return {
            'sum': sum(values),
            'min': Math.min.apply(null, values),
            'max': Math.max.apply(null, values),
            'count': values.length,
            'sumsqr': (function() {
            var sumsqr = 0;
            values.forEach(function (value) {
                sumsqr += value * value;
            });
            return sumsqr;
            })(),
        }
    }
}

The _stats function will also work with “pre-aggregated” values from a map phase. A map function that emits an object containing sum, min, max, count, and sumsqr keys and numeric values for each can use the _stats function to combine these results with the data from other documents. The emitted object may contain other keys (these are ignored by the reducer), and it is also possible to mix raw numeric values and pre-aggregated objects in a single view and obtain the correct aggregated statistics.

Finally, _stats can operate on key-value pairs where each value is an array comprised of numbers or pre-aggregated objects. In this case every value emitted from the map function must be an array, and the arrays must all be the same length, as _stats will compute the statistical quantities above independently for each element in the array. Users who want to compute statistics on multiple values from a single document should either emit each value into the index separately, or compute the statistics for the set of values using the JavaScript example above and emit a pre-aggregated object.

In its simplest variation, _sum sums the numeric values associated with each key, as in the following JavaScript:

// could be replaced by _sum
function(keys, values) {
    return sum(values);
}

As with _stats, the _sum function offers a number of extended capabilities. The _sum function requires that map values be numbers, arrays of numbers, or objects. When presented with array output from a map function, _sum will compute the sum for every element of the array. A bare numeric value will be treated as an array with a single element, and arrays with fewer elements will be treated as if they contained zeroes for every additional element in the longest emitted array. As an example, consider the following map output:

{"total_rows":5, "offset":0, "rows": [
    {"id":"id1", "key":"abc", "value": 2},
    {"id":"id2", "key":"abc", "value": [3,5,7]},
    {"id":"id2", "key":"def", "value": [0,0,0,42]},
    {"id":"id2", "key":"ghi", "value": 1},
    {"id":"id1", "key":"ghi", "value": 3}
]}

The _sum for this output without any grouping would be:

{"rows": [
    {"key":null, "value": [9,5,7,42]}
]}

while the grouped output would be

{"rows": [
    {"key":"abc", "value": [5,5,7]},
    {"key":"def", "value": [0,0,0,42]},
    {"key":"ghi", "value": 4
]}

This is in contrast to the behavior of the _stats function which requires that all emitted values be arrays of identical length if any array is emitted.

It is also possible to have _sum recursively descend through an emitted object and compute the sums for every field in the object. Objects cannot be mixed with other data structures. Objects can be arbitrarily nested, provided that the values for all fields are themselves numbers, arrays of numbers, or objects.

NOTE:

Show Functions

Show functions are used to represent documents in various formats, commonly as HTML pages with nice formatting. They can also be used to run server-side functions without requiring a pre-existing document.

Basic example of show function could be:

function(doc, req){
    if (doc) {
        return "Hello from " + doc._id + "!";
    } else {
        return "Hello, world!";
    }
}

Also, there is more simple way to return json encoded data:

function(doc, req){
    return {
        'json': {
            'id': doc['_id'],
            'rev': doc['_rev']
        }
    }
}

and even files (this one is CouchDB logo):

function(doc, req){
    return {
        'headers': {
            'Content-Type' : 'image/png',
        },
        'base64': ''.concat(
            'iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAMAAAAoLQ9TAAAAsV',
            'BMVEUAAAD////////////////////////5ur3rEBn////////////////wDBL/',
            'AADuBAe9EB3IEBz/7+//X1/qBQn2AgP/f3/ilpzsDxfpChDtDhXeCA76AQH/v7',
            '/84eLyWV/uc3bJPEf/Dw/uw8bRWmP1h4zxSlD6YGHuQ0f6g4XyQkXvCA36MDH6',
            'wMH/z8/yAwX64ODeh47BHiv/Ly/20dLQLTj98PDXWmP/Pz//39/wGyJ7Iy9JAA',
            'AADHRSTlMAbw8vf08/bz+Pv19jK/W3AAAAg0lEQVR4Xp3LRQ4DQRBD0QqTm4Y5',
            'zMxw/4OleiJlHeUtv2X6RbNO1Uqj9g0RMCuQO0vBIg4vMFeOpCWIWmDOw82fZx',
            'vaND1c8OG4vrdOqD8YwgpDYDxRgkSm5rwu0nQVBJuMg++pLXZyr5jnc1BaH4GT',
            'LvEliY253nA3pVhQqdPt0f/erJkMGMB8xucAAAAASUVORK5CYII=')
    }
}

But what if you need to represent data in different formats via a single function? Functions registerType() and provides() are your the best friends in that question:

function(doc, req){
    provides('json', function(){
        return {'json': doc}
    });
    provides('html', function(){
        return '<pre>' + toJSON(doc) + '</pre>'
    })
    provides('xml', function(){
        return {
            'headers': {'Content-Type': 'application/xml'},
            'body' : ''.concat(
                '<?xml version="1.0" encoding="utf-8"?>\n',
                '<doc>',
                (function(){
                    escape = function(s){
                        return s.replace(/&quot;/g, '"')
                                .replace(/&gt;/g, '>')
                                .replace(/&lt;/g, '<')
                                .replace(/&amp;/g, '&');
                    };
                    var content = '';
                    for(var key in doc){
                        if(!doc.hasOwnProperty(key)) continue;
                        var value = escape(toJSON(doc[key]));
                        var key = escape(key);
                        content += ''.concat(
                            '<' + key + '>',
                            value
                            '</' + key + '>'
                        )
                    }
                    return content;
                })(),
                '</doc>'
            )
        }
    })
    registerType('text-json', 'text/json')
    provides('text-json', function(){
        return toJSON(doc);
    })
}

This function may return html, json , xml or our custom text json format representation of same document object with same processing rules. Probably, the xml provider in our function needs more care to handle nested objects correctly, and keys with invalid characters, but you’ve got the idea!

SEE ALSO:

List Functions

While Show Functions are used to customize document presentation, List Functions are used for the same purpose, but on View Functions results.

The following list function formats the view and represents it as a very simple HTML page:

function(head, req){
    start({
        'headers': {
            'Content-Type': 'text/html'
        }
    });
    send('<html><body><table>');
    send('<tr><th>ID</th><th>Key</th><th>Value</th></tr>');
    while(row = getRow()){
        send(''.concat(
            '<tr>',
            '<td>' + toJSON(row.id) + '</td>',
            '<td>' + toJSON(row.key) + '</td>',
            '<td>' + toJSON(row.value) + '</td>',
            '</tr>'
        ));
    }
    send('</table></body></html>');
}

Templates and styles could obviously be used to present data in a nicer fashion, but this is an excellent starting point. Note that you may also use registerType() and provides() functions in a similar way as for Show Functions! However, note that provides() expects the return value to be a string when used inside a list function, so you’ll need to use start() to set any custom headers and stringify your JSON before returning it.

SEE ALSO:

Update Functions

Update handlers are functions that clients can request to invoke server-side logic that will create or update a document. This feature allows a range of use cases such as providing a server-side last modified timestamp, updating individual fields in a document without first getting the latest revision, etc.

When the request to an update handler includes a document ID in the URL, the server will provide the function with the most recent version of that document. You can provide any other values needed by the update handler function via the POST/PUT entity body or query string parameters of the request.

A basic example that demonstrates all use-cases of update handlers:

function(doc, req){
    if (!doc){
        if ('id' in req && req['id']){
            // create new document
            return [{'_id': req['id']}, 'New World']
        }
        // change nothing in database
        return [null, 'Empty World']
    }
    doc['world'] = 'hello';
    doc['edited_by'] = req['userCtx']['name']
    return [doc, 'Edited World!']
}

Filter Functions

Filter functions mostly act like Show Functions and List Functions: they format, or filter the changes feed.

Classic Filters

Filters are special design document functions that allow the changes feed to emit only specific documents that pass filter rules.

Let’s assume that our database is a mailbox and we need to handle only new mail events (documents with the status new). Our filter function would look like this:

function(doc, req){
    // we need only `mail` documents
    if (doc.type != 'mail'){
        return false;
    }
    // we're interested only in `new` ones
    if (doc.status != 'new'){
        return false;
    }
    return true; // passed!
}

Filter functions must return true if a document passed all the rules. Now, if you apply this function to the changes feed it will emit only changes about “new mails”:

GET /somedatabase/_changes?filter=mailbox/new_mail HTTP/1.1

{"results":[
{"seq":"1-g1AAAAF9eJzLYWBg4MhgTmHgz8tPSTV0MDQy1zMAQsMcoARTIkOS_P___7MymBMZc4EC7MmJKSmJqWaYynEakaQAJJPsoaYwgE1JM0o1TjQ3T2HgLM1LSU3LzEtNwa3fAaQ_HqQ_kQG3qgSQqnoCqvJYgCRDA5ACKpxPWOUCiMr9hFUegKi8T1jlA4hKkDuzAC2yZRo","id":"df8eca9da37dade42ee4d7aa3401f1dd","changes":[{"rev":"1-c2e0085a21d34fa1cecb6dc26a4ae657"}]},
{"seq":"9-g1AAAAIreJyVkEsKwjAURUMrqCOXoCuQ5MU0OrI70XyppcaRY92J7kR3ojupaSPUUgqWwAu85By4t0AITbJYo5k7aUNSAnyJ_SGFf4gEkvOyLPMsFtHRL8ZKaC1M0v3eq5ALP-X2a0G1xYKhgnONpmenjT04o_v5tOJ3LV5itTES_uP3FX9ppcAACaVsQAo38hNd_eVFt8ZklVljPqSPYLoH06PJhG0Cxq7-yhQcz-B4_fQCjFuqBjjewVF3E9cORoExSrpU_gHBTo5m","id":"df8eca9da37dade42ee4d7aa34024714","changes":[{"rev":"1-29d748a6e87b43db967fe338bcb08d74"}]},
],
"last_seq":"10-g1AAAAIreJyVkEsKwjAURR9tQR25BF2B5GMaHdmdaNIk1FLjyLHuRHeiO9Gd1LQRaimFlsALvOQcuLcAgGkWKpjbs9I4wYSvkDu4cA-BALkoyzLPQhGc3GKSCqWEjrvfexVy6abc_SxQWwzRVHCuYHaxSpuj1aqfTyp-3-IlSrdakmH8oeKvrRSIkJhSNiKFjdyEm7uc6N6YTKo3iI_pw5se3vRsMiETE23WgzJ5x8s73n-9EMYNTUc4Pt5RdxPVDkYJYxR3qfwLwW6OZw"}

Note that the value of last_seq is 10-.., but we received only two records. Seems like any other changes were for documents that haven’t passed our filter.

We probably need to filter the changes feed of our mailbox by more than a single status value. We’re also interested in statuses like “spam” to update spam-filter heuristic rules, “outgoing” to let a mail daemon actually send mails, and so on. Creating a lot of similar functions that actually do similar work isn’t good idea - so we need a dynamic filter.

You may have noticed that filter functions take a second argument named request. This allows the creation of dynamic filters based on query parameters, user context and more.

The dynamic version of our filter looks like this:

function(doc, req){
    // we need only `mail` documents
    if (doc.type != 'mail'){
        return false;
    }
    // we're interested only in requested status
    if (doc.status != req.query.status){
        return false;
    }
    return true; // passed!
}

and now we have passed the status query parameter in the request to let our filter match only the required documents:

GET /somedatabase/_changes?filter=mailbox/by_status&status=new HTTP/1.1

{"results":[
{"seq":"1-g1AAAAF9eJzLYWBg4MhgTmHgz8tPSTV0MDQy1zMAQsMcoARTIkOS_P___7MymBMZc4EC7MmJKSmJqWaYynEakaQAJJPsoaYwgE1JM0o1TjQ3T2HgLM1LSU3LzEtNwa3fAaQ_HqQ_kQG3qgSQqnoCqvJYgCRDA5ACKpxPWOUCiMr9hFUegKi8T1jlA4hKkDuzAC2yZRo","id":"df8eca9da37dade42ee4d7aa3401f1dd","changes":[{"rev":"1-c2e0085a21d34fa1cecb6dc26a4ae657"}]},
{"seq":"9-g1AAAAIreJyVkEsKwjAURUMrqCOXoCuQ5MU0OrI70XyppcaRY92J7kR3ojupaSPUUgqWwAu85By4t0AITbJYo5k7aUNSAnyJ_SGFf4gEkvOyLPMsFtHRL8ZKaC1M0v3eq5ALP-X2a0G1xYKhgnONpmenjT04o_v5tOJ3LV5itTES_uP3FX9ppcAACaVsQAo38hNd_eVFt8ZklVljPqSPYLoH06PJhG0Cxq7-yhQcz-B4_fQCjFuqBjjewVF3E9cORoExSrpU_gHBTo5m","id":"df8eca9da37dade42ee4d7aa34024714","changes":[{"rev":"1-29d748a6e87b43db967fe338bcb08d74"}]},
],
"last_seq":"10-g1AAAAIreJyVkEsKwjAURR9tQR25BF2B5GMaHdmdaNIk1FLjyLHuRHeiO9Gd1LQRaimFlsALvOQcuLcAgGkWKpjbs9I4wYSvkDu4cA-BALkoyzLPQhGc3GKSCqWEjrvfexVy6abc_SxQWwzRVHCuYHaxSpuj1aqfTyp-3-IlSrdakmH8oeKvrRSIkJhSNiKFjdyEm7uc6N6YTKo3iI_pw5se3vRsMiETE23WgzJ5x8s73n-9EMYNTUc4Pt5RdxPVDkYJYxR3qfwLwW6OZw"}

and we can easily change filter behavior with:

GET /somedatabase/_changes?filter=mailbox/by_status&status=spam HTTP/1.1

{"results":[
{"seq":"6-g1AAAAIreJyVkM0JwjAYQD9bQT05gk4gaWIaPdlNNL_UUuPJs26im-gmuklMjVClFFoCXyDJe_BSAsA4jxVM7VHpJEswWyC_ktJfRBzEzDlX5DGPDv5gJLlSXKfN560KMfdTbL4W-FgM1oQzpmByskqbvdWqnc8qfvvHCyTXWuBu_K7iz38VCOOUENqjwg79hIvfvOhamQahROoVYn3-I5huwXSvm5BJsTbLTk3B8QiO58-_YMoMkT0cr-BwdRElmFKSNKniDcAcjmM","id":"8960e91220798fc9f9d29d24ed612e0d","changes":[{"rev":"3-cc6ff71af716ddc2ba114967025c0ee0"}]},
],
"last_seq":"10-g1AAAAIreJyVkEsKwjAURR9tQR25BF2B5GMaHdmdaNIk1FLjyLHuRHeiO9Gd1LQRaimFlsALvOQcuLcAgGkWKpjbs9I4wYSvkDu4cA-BALkoyzLPQhGc3GKSCqWEjrvfexVy6abc_SxQWwzRVHCuYHaxSpuj1aqfTyp-3-IlSrdakmH8oeKvrRSIkJhSNiKFjdyEm7uc6N6YTKo3iI_pw5se3vRsMiETE23WgzJ5x8s73n-9EMYNTUc4Pt5RdxPVDkYJYxR3qfwLwW6OZw"}

Combining filters with a continuous feed allows creating powerful event-driven systems.

View Filters

To use them just pass filter=_view and view=designdoc/viewname as request parameters to the changes feed:

GET /somedatabase/_changes?filter=_view&view=dname/viewname  HTTP/1.1

NOTE:

SEE ALSO:

Validate Document Update Functions

A design document may contain a function named validate_doc_update which can be used to prevent invalid or unauthorized document update requests from being stored. The function is passed the new document from the update request, the current document stored in the database, a userctx_object containing information about the user writing the document (if present), and a security_object with lists of database security roles.

Validation functions typically examine the structure of the new document to ensure that required fields are present and to verify that the requesting user should be allowed to make changes to the document properties. For example, an application may require that a user must be authenticated in order to create a new document or that specific document fields be present when a document is updated. The validation function can abort the pending document write by throwing one of two error objects:

// user is not authorized to make the change but may re-authenticate
throw({ unauthorized: 'Error message here.' });
// change is not allowed
throw({ forbidden: 'Error message here.' });

Document validation is optional, and each design document in the database may have at most one validation function. When a write request is received for a given database, the validation function in each design document in that database is called in an unspecified order. If any of the validation functions throw an error, the write will not succeed.

Example: The _design/_auth ddoc from _users database uses a validation function to ensure that documents contain some required fields and are only modified by a user with the _admin role:

function(newDoc, oldDoc, userCtx, secObj) {
    if (newDoc._deleted === true) {
        // allow deletes by admins and matching users
        // without checking the other fields
        if ((userCtx.roles.indexOf('_admin') !== -1) ||
            (userCtx.name == oldDoc.name)) {
            return;
        } else {
            throw({forbidden: 'Only admins may delete other user docs.'});
        }
    }
    if ((oldDoc && oldDoc.type !== 'user') || newDoc.type !== 'user') {
        throw({forbidden : 'doc.type must be user'});
    } // we only allow user docs for now
    if (!newDoc.name) {
        throw({forbidden: 'doc.name is required'});
    }
    if (!newDoc.roles) {
        throw({forbidden: 'doc.roles must exist'});
    }
    if (!isArray(newDoc.roles)) {
        throw({forbidden: 'doc.roles must be an array'});
    }
    if (newDoc._id !== ('org.couchdb.user:' + newDoc.name)) {
        throw({
            forbidden: 'Doc ID must be of the form org.couchdb.user:name'
        });
    }
    if (oldDoc) { // validate all updates
        if (oldDoc.name !== newDoc.name) {
            throw({forbidden: 'Usernames can not be changed.'});
        }
    }
    if (newDoc.password_sha && !newDoc.salt) {
        throw({
            forbidden: 'Users with password_sha must have a salt.' +
                'See /_utils/script/couch.js for example code.'
        });
    }
    var is_server_or_database_admin = function(userCtx, secObj) {
        // see if the user is a server admin
        if(userCtx.roles.indexOf('_admin') !== -1) {
            return true; // a server admin
        }
        // see if the user a database admin specified by name
        if(secObj && secObj.admins && secObj.admins.names) {
            if(secObj.admins.names.indexOf(userCtx.name) !== -1) {
                return true; // database admin
            }
        }
        // see if the user a database admin specified by role
        if(secObj && secObj.admins && secObj.admins.roles) {
            var db_roles = secObj.admins.roles;
            for(var idx = 0; idx < userCtx.roles.length; idx++) {
                var user_role = userCtx.roles[idx];
                if(db_roles.indexOf(user_role) !== -1) {
                    return true; // role matches!
                }
            }
        }
        return false; // default to no admin
    }
    if (!is_server_or_database_admin(userCtx, secObj)) {
        if (oldDoc) { // validate non-admin updates
            if (userCtx.name !== newDoc.name) {
                throw({
                    forbidden: 'You may only update your own user document.'
                });
            }
            // validate role updates
            var oldRoles = oldDoc.roles.sort();
            var newRoles = newDoc.roles.sort();
            if (oldRoles.length !== newRoles.length) {
                throw({forbidden: 'Only _admin may edit roles'});
            }
            for (var i = 0; i < oldRoles.length; i++) {
                if (oldRoles[i] !== newRoles[i]) {
                    throw({forbidden: 'Only _admin may edit roles'});
                }
            }
        } else if (newDoc.roles.length > 0) {
            throw({forbidden: 'Only _admin may set roles'});
        }
    }
    // no system roles in users db
    for (var i = 0; i < newDoc.roles.length; i++) {
        if (newDoc.roles[i][0] === '_') {
            throw({
                forbidden:
                'No system roles (starting with underscore) in users db.'
            });
        }
    }
    // no system names as names
    if (newDoc.name[0] === '_') {
        throw({forbidden: 'Username may not start with underscore.'});
    }
    var badUserNameChars = [':'];
    for (var i = 0; i < badUserNameChars.length; i++) {
        if (newDoc.name.indexOf(badUserNameChars[i]) >= 0) {
            throw({forbidden: 'Character `' + badUserNameChars[i] +
                    '` is not allowed in usernames.'});
        }
    }
}

NOTE:

SEE ALSO:

Guide to Views

Introduction to Views

What Is a View?

{
    "_id":"biking",
    "_rev":"AE19EBC7654",
    "title":"Biking",
    "body":"My biggest hobby is mountainbiking. The other day...",
    "date":"2009/01/30 18:04:11"
}

{
    "_id":"bought-a-cat",
    "_rev":"4A3BBEE711",
    "title":"Bought a Cat",
    "body":"I went to the the pet store earlier and brought home a little kitty...",
    "date":"2009/02/17 21:13:39"
}

{
    "_id":"hello-world",
    "_rev":"43FBA4E7AB",
    "title":"Hello World",
    "body":"Well hello and welcome to my new blog...",
    "date":"2009/01/15 15:52:20"
}

Three will do for the example. Note that the documents are sorted by “_id”, which is how they are stored in the database. Now we define a view. Bear with us without an explanation while we show you some code:

function(doc) {
    if(doc.date && doc.title) {
        emit(doc.date, doc.title);
    }
}

This is a map function, and it is written in JavaScript. If you are not familiar with JavaScript but have used C or any other C-like language such as Java, PHP, or C#, this should look familiar. It is a simple function definition.

You provide CouchDB with view functions as strings stored inside the views field of a design document. To create this view you can use this command:

curl -X PUT http://admin:password@127.0.0.1:5984/db/_design/my_ddoc
     -d '{"views":{"my_filter":{"map":
         "function(doc) { if(doc.date && doc.title) { emit(doc.date, doc.title); }}"}}}'

You don’t run the JavaScript function yourself. Instead, when you query your view, CouchDB takes the source code and runs it for you on every document in the database your view was defined in. You query your view to retrieve the view result using the following command:

curl -X GET http://admin:password@127.0.0.1:5984/db/_design/my_ddoc/_view/my_filter

All map functions have a single parameter doc. This is a single document in the database. Our map function checks whether our document has a date and a title attribute — luckily, all of our documents have them — and then calls the built-in emit() function with these two attributes as arguments.

The emit() function always takes two arguments: the first is key, and the second is value. The emit(key, value) function creates an entry in our view result. One more thing: the emit() function can be called multiple times in the map function to create multiple entries in the view results from a single document, but we are not doing that yet.

CouchDB takes whatever you pass into the emit() function and puts it into a list (see Table 1, “View results” below). Each row in that list includes the key and value. More importantly, the list is sorted by key (by doc.date in our case). The most important feature of a view result is that it is sorted by key. We will come back to that over and over again to do neat things. Stay tuned.

Table 1. View results:

When you query your view, CouchDB takes the source code and runs it for you on every document in the database. If you have a lot of documents, that takes quite a bit of time and you might wonder if it is not horribly inefficient to do this. Yes, it would be, but CouchDB is designed to avoid any extra costs: it only runs through all documents once, when you first query your view. If a document is changed, the map function is only run once, to recompute the keys and values for that single document.

The view result is stored in a B-tree, just like the structure that is responsible for holding your documents. View B-trees are stored in their own file, so that for high-performance CouchDB usage, you can keep views on their own disk. The B-tree provides very fast lookups of rows by key, as well as efficient streaming of rows in a key range. In our example, a single view can answer all questions that involve time: “Give me all the blog posts from last week” or “last month” or “this year.” Pretty neat.

When we query our view, we get back a list of all documents sorted by date. Each row also includes the post title so we can construct links to posts. Table 1 is just a graphical representation of the view result. The actual result is JSON-encoded and contains a little more metadata:

{
    "total_rows": 3,
    "offset": 0,
    "rows": [
        {
            "key": "2009/01/15 15:52:20",
            "id": "hello-world",
            "value": "Hello World"
        },
        {
            "key": "2009/01/30 18:04:11",
            "id": "biking",
            "value": "Biking"
        },
        {
            "key": "2009/02/17 21:13:39",
            "id": "bought-a-cat",
            "value": "Bought a Cat"
        }
    ]
}

Now, the actual result is not as nicely formatted and doesn’t include any superfluous whitespace or newlines, but this is better for you (and us!) to read and understand. Where does that “id” member in the result rows come from? That wasn’t there before. That’s because we omitted it earlier to avoid confusion. CouchDB automatically includes the document ID of the document that created the entry in the view result. We’ll use this as well when constructing links to the blog post pages.

WARNING:

Efficient Lookups

First, back to the B-trees! We explained that the B-tree that backs the key-sorted view result is built only once, when you first query a view, and all subsequent queries will just read the B-tree instead of executing the map function for all documents again. What happens, though, when you change a document, add a new one, or delete one? Easy: CouchDB is smart enough to find the rows in the view result that were created by a specific document. It marks them invalid so that they no longer show up in view results. If the document was deleted, we’re good — the resulting B-tree reflects the state of the database. If a document got updated, the new document is run through the map function and the resulting new lines are inserted into the B-tree at the correct spots. New documents are handled in the same way. The B-tree is a very efficient data structure for our needs, and the crash-only design of CouchDB databases is carried over to the view indexes as well.

To add one more point to the efficiency discussion: usually multiple documents are updated between view queries. The mechanism explained in the previous paragraph gets applied to all changes in the database since the last time the view was queried in a batch operation, which makes things even faster and is generally a better use of your resources.

Find One

Note that in the case where multiple rows have the same key (perhaps we design a view where the key is the name of the post’s author), key queries can return more than one row.

Find Many

/blog/_design/docs/_view/by_date?startkey="2010/01/01 00:00:00"&endkey="2010/02/00 00:00:00"

The startkey and endkey parameters specify an inclusive range on which we can search.

To make things a little nicer and to prepare for a future example, we are going to change the format of our date field. Instead of a string, we are going to use an array, where individual members are part of a timestamp in decreasing significance. This sounds fancy, but it is rather easy. Instead of:

{
    "date": "2009/01/31 00:00:00"
}

we use:

{
    "date": [2009, 1, 31, 0, 0, 0]
}

Our map function does not have to change for this, but our view result looks a little different:

Table 2. New view results:

And our queries change to:

/blog/_design/docs/_view/by_date?startkey=[2010, 1, 1, 0, 0, 0]&endkey=[2010, 2, 1, 0, 0, 0]

For all you care, this is just a change in syntax, not meaning. But it shows you the power of views. Not only can you construct an index with scalar values like strings and integers, you can also use JSON structures as keys for your views. Say we tag our documents with a list of tags and want to see all tags, but we don’t care for documents that have not been tagged.

{
    ...
    tags: ["cool", "freak", "plankton"],
    ...
}

{
    ...
    tags: [],
    ...
}

function(doc) {
    if(doc.tags.length > 0) {
        for(var idx in doc.tags) {
            emit(doc.tags[idx], null);
        }
    }
}

This shows a few new things. You can have conditions on structure (if(doc.tags.length > 0)) instead of just values. This is also an example of how a map function calls emit() multiple times per document. And finally, you can pass null instead of a value to the value parameter. The same is true for the key parameter. We’ll see in a bit how that is useful.

Reversed Results

It’s pretty easy to understand when you see how view query options work under the hood. A view is stored in a tree structure for fast lookups. Whenever you query a view, this is how CouchDB operates:

If you specify descending=true, the reading direction is reversed, not the sort order of the rows in the view. In addition, the same two-step procedure is followed.

Say you have a view result that looks like this:

Here are potential query options: ?startkey=1&descending=true. What will CouchDB do? See #1 above: it jumps to startkey, which is the row with the key 1, and starts reading backward until it hits the end of the view. So the particular result would be:

This is very likely not what you want. To get the rows with the indexes 1 and 2 in reverse order, you need to switch the startkey to endkey: endkey=1&descending=true:

Now that looks a lot better. CouchDB started reading at the bottom of the view and went backward until it hit endkey.

The View to Get Comments for Posts

In the blog app, we use group_level reduce queries to compute the count of comments both on a per-post and total basis, achieved by querying the same view index with different methods. With some array keys, and assuming each key has the value 1:

["a","b","c"]
["a","b","e"]
["a","c","m"]
["b","a","c"]
["b","a","g"]

the reduce view:

function(keys, values, rereduce) {
    return sum(values)
}

or:

_sum

which is a built-in CouchDB reduce function (the others are _count and _stats). _sum here returns the total number of rows between the start and end key. So with startkey=["a","b"]&endkey=["b"] (which includes the first three of the above keys) the result would equal 3. The effect is to count rows. If you’d like to count rows without depending on the row value, you can switch on the rereduce parameter:

function(keys, values, rereduce) {
    if (rereduce) {
        return sum(values);
    } else {
        return values.length;
    }
}

NOTE:

["a"]   3
["b"]   2

["a","b"]   2
["a","c"]   1
["b","a"]   2

Reduce/Rereduce

Consider the map result are:

"afrikaans", 1
"afrikaans", 1
"chinese", 1
"chinese", 1
"chinese", 1
"chinese", 1
"french", 1
"italian", 1
"italian", 1
"spanish", 1
"vietnamese", 1
"vietnamese", 1

Example 1. Example view result (mmm, food)

When we want to find out how many dishes there are per origin, we can reuse the simple reduce function shown earlier:

function(keys, values, rereduce) {
    return sum(values);
}

Figure 2, “The B-tree index” shows a simplified version of what the B-tree index looks like. We abbreviated the key strings.

function(null, [3, 1], true) {
    return sum([3, 1]);
}

"abc", "afrikaans"
"cef", "afrikaans"
"fhi", "chinese"
"hkl", "chinese"
"ino", "chinese"
"lqr", "chinese"
"mtu", "french"
"owx", "italian"
"qza", "italian"
"tdx", "spanish"
"xfg", "vietnamese"
"zul", "vietnamese"

function(keys, values, rereduce) {
    var unique_labels = {};
    values.forEach(function(label) {
        if(!unique_labels[label]) {
            unique_labels[label] = true;
        }
    });
    return unique_labels;
}

One vs. Multiple Design Documents

Each view you create corresponds to one B-tree. All views in a single design document will live in the same set of index files on disk (one file per database shard; in 2.0+ by default, 8 files per node).

The most practical consideration for separating views into separate documents is how often you change those views. Views that change often, and are in the same design document as other views, will invalidate those other views’ indexes when the design document is written, forcing them all to rebuild from scratch. Obviously you will want to avoid this in production!

However, when you have multiple views with the same map function in the same design document, CouchDB will optimize and only calculate that map function once. This lets you have two views with different reduce functions (say, one with _sum and one with _stats) but build only a single copy of the mapped index. It also saves disk space and the time to write multiple copies to disk.

Another benefit of having multiple views in the same design document is that the index files can keep a single index of backwards references from docids to rows. CouchDB needs these “back refs” to invalidate rows in a view when a document is deleted (otherwise, a delete would force a total rebuild!)

One other consideration is that each separate design document will spawn another (set of) couchjs processes to generate the view, one per shard. Depending on the number of cores on your server(s), this may be efficient (using all of the idle cores you have) or inefficient (overloading the CPU on your servers). The exact situation will depend on your deployment architecture.

So, should you use one or multiple design documents? The choice is yours.

Lessons Learned

Wrapping Up

Reduce functions operate on the sorted rows emitted by map view functions. CouchDB’s reduce functionality takes advantage of one of the fundamental properties of B-tree indexes: for every leaf node (a sorted row), there is a chain of internal nodes reaching back to the root. Each leaf node in the B-tree carries a few rows (on the order of tens, depending on row size), and each internal node may link to a few leaf nodes or other internal nodes.

The reduce function is run on every node in the tree in order to calculate the final reduce value. The end result is a reduce function that can be incrementally updated upon changes to the map function, while recalculating the reduction values for a minimum number of nodes. The initial reduction is calculated once per each node (inner and leaf) in the tree.

When run on leaf nodes (which contain actual map rows), the reduce function’s third parameter, rereduce, is false. The arguments in this case are the keys and values as output by the map function. The function has a single returned reduction value, which is stored on the inner node that a working set of leaf nodes have in common, and is used as a cache in future reduce calculations.

When the reduce function is run on inner nodes, the rereduce flag is true. This allows the function to account for the fact that it will be receiving its own prior output. When rereduce is true, the values passed to the function are intermediate reduction values as cached from previous calculations. When the tree is more than two levels deep, the rereduce phase is repeated, consuming chunks of the previous level’s output until the final reduce value is calculated at the root node.

A common mistake new CouchDB users make is attempting to construct complex aggregate values with a reduce function. Full reductions should result in a scalar value, like 5, and not, for instance, a JSON hash with a set of unique keys and the count of each. The problem with this approach is that you’ll end up with a very large final value. The number of unique keys can be nearly as large as the number of total keys, even for a large set. It is fine to combine a few scalar calculations into one reduce function; for instance, to find the total, average, and standard deviation of a set of numbers in a single function.

If you’re interested in pushing the edge of CouchDB’s incremental reduce functionality, have a look at Google’s paper on Sawzall, which gives examples of some of the more exotic reductions that can be accomplished in a system with similar constraints.

Views Collation

Basics

function(doc) {
    if (doc.Type == "customer") {
        emit(doc.LastName, {FirstName: doc.FirstName, Address: doc.Address});
    }
}

CouchDB allows arbitrary JSON structures to be used as keys. You can use JSON arrays as keys for fine-grained control over sorting and grouping.

Examples

function(doc) {
    if (doc.Type == "customer") {
        emit([doc._id, 0], null);
    } else if (doc.Type == "order") {
        emit([doc.customer_id, 1], null);
    }
}

To list a specific customer with _id XYZ, and all of that customer’s orders, limit the startkey and endkey ranges to cover only documents for that customer’s _id:

startkey=["XYZ"]&endkey=["XYZ", {}]

It is not recommended to emit the document itself in the view. Instead, to include the bodies of the documents when requesting the view, request the view with ?include_docs=true.

Sorting by Dates

emit(Date.parse(doc.created_at).getTime(), null);

Alternatively, if you use a date format which sorts lexicographically, such as "2013/06/09 13:52:11 +0000" you can just

emit(doc.created_at, null);

and avoid the conversion. As a bonus, this date format is compatible with the JavaScript date parser, so you can use new Date(doc.created_at) in your client side JavaScript to make date sorting easy in the browser.

String Ranges

That is, rather than:

startkey="abc"&endkey="abcZZZZZZZZZ"

You should use:

startkey="abc"&endkey="abc\ufff0"

Collation Specification

// special values sort before all other types
null
false
true
// then numbers
1
2
3.0
4
// then text, case sensitive
"a"
"A"
"aa"
"b"
"B"
"ba"
"bb"
// then arrays. compared element by element until different.
// Longer arrays sort after their prefixes
["a"]
["b"]
["b","c"]
["b","c", "a"]
["b","d"]
["b","d", "e"]
// then object, compares each key value in the list until different.
// larger objects sort after their subset objects.
{a:1}
{a:2}
{b:1}
{b:2}
{b:2, a:1} // Member order does matter for collation.
           // CouchDB preserves member order
           // but doesn't require that clients will.
           // this test might fail if used with a js engine
           // that doesn't preserve order
{b:2, c:2}

Comparison of strings is done using ICU which implements the Unicode Collation Algorithm, giving a dictionary sorting of keys. This can give surprising results if you were expecting ASCII ordering. Note that:

You can demonstrate the collation sequence for 7-bit ASCII characters like this:

require 'rubygems'
require 'restclient'
require 'json'
DB="http://127.0.0.1:5984/collator"
RestClient.delete DB rescue nil
RestClient.put "#{DB}",""
(32..126).each do |c|
    RestClient.put "#{DB}/#{c.to_s(16)}", {"x"=>c.chr}.to_json
end
RestClient.put "#{DB}/_design/test", <<EOS
{
    "views":{
        "one":{
            "map":"function (doc) { emit(doc.x,null); }"
        }
    }
}
EOS
puts RestClient.get("#{DB}/_design/test/_view/one")

This shows the collation sequence to be:

` ^ _ - , ; : ! ? . ' " ( ) [ ] { } @ * / \ & # % + < = > | ~ $ 0 1 2 3 4 5 6 7 8 9
a A b B c C d D e E f F g G h H i I j J k K l L m M n N o O p P q Q r R s S t T u U v V w W x X y Y z Z

Key ranges

startkey="Abc"&endkey="AbcZZZZ"

will match “ABC” and “abc1”, but not “abc”. This is because UCA sorts as:

abc < Abc < ABC < abc1 < AbcZZZZZ

For most applications, to avoid problems you should lowercase the startkey:

startkey="abc"&endkey="abcZZZZZZZZ"

will match all keys starting with [aA][bB][cC]

Complex keys

_all_docs

startkey="_design/"&endkey="_design/ZZZZZZZZ"

will not find _design/abc because ‘Z’ comes before ‘a’ in the ASCII sequence. A better solution is:

startkey="_design/"&endkey="_design0"

Raw collation

1
false
null
true
{"a":"a"},
["a"]
"a"

Beware that {} is no longer a suitable “high” key sentinel value. Use a string like "\ufff0" instead.

Joins With Views

Linked Documents

This means that if one document contains the ids of other documents, it can cause those documents to be fetched in the view too, adjacent to the same key if required.

For example, if you have the following hierarchically-linked documents:

[
    { "_id": "11111" },
    { "_id": "22222", "ancestors": ["11111"], "value": "hello" },
    { "_id": "33333", "ancestors": ["22222","11111"], "value": "world" }
]

You can emit the values with the ancestor documents adjacent to them in the view like this:

function(doc) {
    if (doc.value) {
        emit([doc.value, 0], null);
        if (doc.ancestors) {
            for (var i in doc.ancestors) {
                emit([doc.value, Number(i)+1], {_id: doc.ancestors[i]});
            }
        }
    }
}

The result you get is:

{
    "total_rows": 5,
    "offset": 0,
    "rows": [
        {
            "id": "22222",
            "key": [
                "hello",
                0
            ],
            "value": null,
            "doc": {
                "_id": "22222",
                "_rev": "1-0eee81fecb5aa4f51e285c621271ff02",
                "ancestors": [
                    "11111"
                ],
                "value": "hello"
            }
        },
        {
            "id": "22222",
            "key": [
                "hello",
                1
            ],
            "value": {
                "_id": "11111"
            },
            "doc": {
                "_id": "11111",
                "_rev": "1-967a00dff5e02add41819138abb3284d"
            }
        },
        {
            "id": "33333",
            "key": [
                "world",
                0
            ],
            "value": null,
            "doc": {
                "_id": "33333",
                "_rev": "1-11e42b44fdb3d3784602eca7c0332a43",
                "ancestors": [
                    "22222",
                    "11111"
                ],
                "value": "world"
            }
        },
        {
            "id": "33333",
            "key": [
                "world",
                1
            ],
            "value": {
                "_id": "22222"
            },
            "doc": {
                "_id": "22222",
                "_rev": "1-0eee81fecb5aa4f51e285c621271ff02",
                "ancestors": [
                    "11111"
                ],
                "value": "hello"
            }
        },
        {
            "id": "33333",
            "key": [
                "world",
                2
            ],
            "value": {
                "_id": "11111"
            },
            "doc": {
                "_id": "11111",
                "_rev": "1-967a00dff5e02add41819138abb3284d"
            }
        }
    ]
}

which makes it very cheap to fetch a document plus all its ancestors in one query.

Note that the "id" in the row is still that of the originating document. The only difference is that include_docs fetches a different doc.

The current revision of the document is resolved at query time, not at the time the view is generated. This means that if a new revision of the linked document is added later, it will appear in view queries even though the view itself hasn’t changed. To force a specific revision of a linked document to be used, emit a "_rev" property as well as "_id".

Using View Collation

Just today, there was a discussion on IRC on how you’d go about modeling a simple blogging system with “post” and “comment” entities, where any blog post might have N comments. If you’d be using an SQL database, you’d obviously have two tables with foreign keys and you’d be using joins. (At least until you needed to add some denormalization).

But what would the “obvious” approach in CouchDB look like?

Approach #1: Comments Inlined

{
    "_id": "myslug",
    "_rev": "123456",
    "author": "john",
    "title": "My blog post",
    "content": "Bla bla bla …",
    "comments": [
        {"author": "jack", "content": "…"},
        {"author": "jane", "content": "…"}
    ]
}

NOTE:

The obvious advantage of this approach is that the data that belongs together is stored in one place. Delete the post, and you automatically delete the corresponding comments, and so on.

You may be thinking that putting the comments inside the blog post document would not allow us to query for the comments themselves, but you’d be wrong. You could trivially write a CouchDB view that would return all comments across all blog posts, keyed by author:

function(doc) {
    for (var i in doc.comments) {
        emit(doc.comments[i].author, doc.comments[i].content);
    }
}

Now you could list all comments by a particular user by invoking the view and passing it a ?key="username" query string parameter.

However, this approach has a drawback that can be quite significant for many applications: To add a comment to a post, you need to:

Now if you have multiple client processes adding comments at roughly the same time, some of them will get a HTTP 409 Conflict error on step 3 (that’s optimistic concurrency in action). For some applications this makes sense, but in many other apps, you’d want to append new related data regardless of whether other data has been added in the meantime.

The only way to allow non-conflicting addition of related data is by putting that related data into separate documents.

Approach #2: Comments Separate

The blog post document would look similar to the above, minus the comments property. Also, we’d now have a type property on all our documents so that we can tell the difference between posts and comments:

{
    "_id": "myslug",
    "_rev": "123456",
    "type": "post",
    "author": "john",
    "title": "My blog post",
    "content": "Bla bla bla …"
}

The comments themselves are stored in separate documents, which also have a type property (this time with the value “comment”), and additionally feature a post property containing the ID of the post document they belong to:

{
    "_id": "ABCDEF",
    "_rev": "123456",
    "type": "comment",
    "post": "myslug",
    "author": "jack",
    "content": "…"
}

{
    "_id": "DEFABC",
    "_rev": "123456",
    "type": "comment",
    "post": "myslug",
    "author": "jane",
    "content": "…"
}

To list all comments per blog post, you’d add a simple view, keyed by blog post ID:

function(doc) {
    if (doc.type == "comment") {
        emit(doc.post, {author: doc.author, content: doc.content});
    }
}

And you’d invoke that view passing it a ?key="post_id" query string parameter.

Viewing all comments by author is just as easy as before:

function(doc) {
    if (doc.type == "comment") {
        emit(doc.author, {post: doc.post, content: doc.content});
    }
}

So this is better in some ways, but it also has a disadvantage. Imagine you want to display a blog post with all the associated comments on the same web page. With our first approach, we needed just a single request to the CouchDB server, namely a GET request to the document. With this second approach, we need two requests: a GET request to the post document, and a GET request to the view that returns all comments for the post.

That is okay, but not quite satisfactory. Just imagine you wanted to add threaded comments: you’d now need an additional fetch per comment. What we’d probably want then would be a way to join the blog post and the various comments together to be able to retrieve them with a single HTTP request.

This was when Damien Katz, the author of CouchDB, chimed in to the discussion on IRC to show us the way.

Optimization: Using the Power of View Collation

function(doc) {
    if (doc.type == "post") {
        emit([doc._id, 0], null);
    } else if (doc.type == "comment") {
        emit([doc.post, 1], null);
    }
}

Okay, this may be confusing at first. Let’s take a step back and look at what views in CouchDB are really about.

CouchDB views are basically highly efficient on-disk dictionaries that map keys to values, where the key is automatically indexed and can be used to filter and/or sort the results you get back from your views. When you “invoke” a view, you can say that you’re only interested in a subset of the view rows by specifying a ?key=foo query string parameter. Or you can specify ?startkey=foo and/or ?endkey=bar query string parameters to fetch rows over a range of keys. Finally, by adding ?include_docs=true to the query, the result will include the full body of each emitted document.

It’s also important to note that keys are always used for collating (i.e. sorting) the rows. CouchDB has well defined (but as of yet undocumented) rules for comparing arbitrary JSON objects for collation. For example, the JSON value ["foo", 2] is sorted after (considered “greater than”) the values ["foo"] or ["foo", 1, "bar"], but before e.g. ["foo", 2, "bar"]. This feature enables a whole class of tricks that are rather non-obvious…

SEE ALSO:

With that in mind, let’s return to the view function above. First note that, unlike the previous view functions we’ve used here, this view handles both “post” and “comment” documents, and both of them end up as rows in the same view. Also, the key in this view is not just a simple string, but an array. The first element in that array is always the ID of the post, regardless of whether we’re processing an actual post document, or a comment associated with a post. The second element is 0 for post documents, and 1 for comment documents.

Let’s assume we have two blog posts in our database. Without limiting the view results via key, startkey, or endkey, we’d get back something like the following:

{
    "total_rows": 5, "offset": 0, "rows": [{
            "id": "myslug",
            "key": ["myslug", 0],
            "value": null
        }, {
            "id": "ABCDEF",
            "key": ["myslug", 1],
            "value": null
        }, {
            "id": "DEFABC",
            "key": ["myslug", 1],
            "value": null
        }, {
            "id": "other_slug",
            "key": ["other_slug", 0],
            "value": null
        }, {
            "id": "CDEFAB",
            "key": ["other_slug", 1],
            "value": null
        },
    ]
}

NOTE:

Now, to get a specific blog post and all associated comments, we’d invoke that view with the query string:

?startkey=["myslug"]&endkey=["myslug", 2]&include_docs=true

We’d get back the first three rows, those that belong to the myslug post, but not the others, along with the full bodies of each document. Et voila, we now have the data we need to display a post with all associated comments, retrieved via a single GET request.

You may be asking what the 0 and 1 parts of the keys are for. They’re simply to ensure that the post document is always sorted before the the associated comment documents. So when you get back the results from this view for a specific post, you’ll know that the first row contains the data for the blog post itself, and the remaining rows contain the comment data.

One remaining problem with this model is that comments are not ordered, but that’s simply because we don’t have date/time information associated with them. If we had, we’d add the timestamp as third element of the key array, probably as ISO date/time strings. Now we would continue using the query string ?startkey=["myslug"]&endkey=["myslug", 2]&include_docs=true to fetch the blog post and all associated comments, only now they’d be in chronological order.

View Cookbook for SQL Jockeys

Using Views

CREATE TABLE

or:

ALTER TABLE

How you can do this in CouchDB?

Using views is a two-step process. First you define a view; then you query it. This is analogous to defining a table structure (with indexes) using CREATE TABLE or ALTER TABLE and querying it using an SQL query.

Defining a View

{
    "_id": "_design/application",
    "_rev": "1-C1687D17",
    "views": {
        "viewname": {
            "map": "function(doc) { ... }",
            "reduce": "function(keys, values) { ... }"
        }
    }
}

We are defining a view viewname. The definition of the view consists of two functions: the map function and the reduce function. Specifying a reduce function is optional. We’ll look at the nature of the functions later. Note that viewname can be whatever you like: users, by-name, or by-date are just some examples.

A single design document can also include multiple view definitions, each identified by a unique name:

{
    "_id": "_design/application",
    "_rev": "1-C1687D17",
    "views": {
        "viewname": {
            "map": "function(doc) { ... }",
            "reduce": "function(keys, values) { ... }"
        },
        "anotherview": {
            "map": "function(doc) { ... }",
            "reduce": "function(keys, values) { ... }"
        }
    }
}

Querying a View

/database/_design/application/_view/viewname

database is the name of the database you created your design document in. Next up is the design document name, and then the view name prefixed with _view/. To query anotherview, replace viewname in that URI with anotherview. If you want to query a view in a different design document, adjust the design document name.

MapReduce Functions

CouchDB is smart enough to run a map function only once for every document, even on subsequent queries on a view. Only changes to documents or new documents need to be processed anew.

Map functions

The map result looks like this:

{"total_rows":3,"offset":0,"rows":[
{"id":"fc2636bf50556346f1ce46b4bc01fe30","key":"Lena","value":5},
{"id":"1fb2449f9b9d4e466dbfa47ebe675063","key":"Lisa","value":4},
{"id":"8ede09f6f6aeb35d948485624b28f149","key":"Sarah","value":6}
]}

It is a list of rows sorted by the value of key. The id is added automatically and refers back to the document that created this row. The value is the data you’re looking for. For example purposes, it’s the girl’s age.

The map function that produces this result is:

function(doc) {
    if(doc.name && doc.age) {
        emit(doc.name, doc.age);
    }
}

It includes the if statement as a sanity check to ensure that we’re operating on the right fields and calls the emit function with the name and age as the key and value.

Look Up by Key

SELECT field FROM table WHERE value="searchterm"

How you can do this in CouchDB?

Use case: get a result (which can be a record or set of records) associated with a key (“searchterm”).

To look something up quickly, regardless of the storage mechanism, an index is needed. An index is a data structure optimized for quick search and retrieval. CouchDB’s map result is stored in such an index, which happens to be a B+ tree.

To look up a value by “searchterm”, we need to put all values into the key of a view. All we need is a simple map function:

function(doc) {
    if(doc.value) {
        emit(doc.value, null);
    }
}

This creates a list of documents that have a value field sorted by the data in the value field. To find all the records that match “searchterm”, we query the view and specify the search term as a query parameter:

/database/_design/application/_view/viewname?key="searchterm"

Consider the documents from the previous section, and say we’re indexing on the age field of the documents to find all the five-year-olds:

function(doc) {
    if(doc.age && doc.name) {
        emit(doc.age, doc.name);
    }
}

Query:

/ladies/_design/ladies/_view/age?key=5

Result:

{"total_rows":3,"offset":1,"rows":[
{"id":"fc2636bf50556346f1ce46b4bc01fe30","key":5,"value":"Lena"}
]}

Easy.

Note that you have to emit a value. The view result includes the associated document ID in every row. We can use it to look up more data from the document itself. We can also use the ?include_docs=true parameter to have CouchDB fetch the individual documents for us.

Look Up by Prefix

SELECT field FROM table WHERE value LIKE "searchterm%"

How you can do this in CouchDB?

Use case: find all documents that have a field value that starts with searchterm. For example, say you stored a MIME type (like text/html or image/jpg) for each document and now you want to find all documents that are images according to the MIME type.

The solution is very similar to the previous example: all we need is a map function that is a little more clever than the first one. But first, an example document:

{
    "_id": "Hugh Laurie",
    "_rev": "1-9fded7deef52ac373119d05435581edf",
    "mime-type": "image/jpg",
    "description": "some dude"
}

The clue lies in extracting the prefix that we want to search for from our document and putting it into our view index. We use a regular expression to match our prefix:

function(doc) {
    if(doc["mime-type"]) {
        // from the start (^) match everything that is not a slash ([^\/]+) until
        // we find a slash (\/). Slashes needs to be escaped with a backslash (\/)
        var prefix = doc["mime-type"].match(/^[^\/]+\//);
        if(prefix) {
          emit(prefix, null);
        }
    }
}

We can now query this view with our desired MIME type prefix and not only find all images, but also text, video, and all other formats:

/files/_design/finder/_view/by-mime-type?key="image/"

Aggregate Functions

SELECT COUNT(field) FROM table

How you can do this in CouchDB?

Use case: calculate a derived value from your data.

We haven’t explained reduce functions yet. Reduce functions are similar to aggregate functions in SQL. They compute a value over multiple documents.

To explain the mechanics of reduce functions, we’ll create one that doesn’t make a whole lot of sense. But this example is easy to understand. We’ll explore more useful reductions later.

Reduce functions operate on the output of the map function (also called the map result or intermediate result). The reduce function’s job, unsurprisingly, is to reduce the list that the map function produces.

Here’s what our summing reduce function looks like:

function(keys, values) {
    var sum = 0;
    for(var idx in values) {
        sum = sum + values[idx];
    }
    return sum;
}

Here’s an alternate, more idiomatic JavaScript version:

function(keys, values) {
    var sum = 0;
    values.forEach(function(element) {
        sum = sum + element;
    });
    return sum;
}

NOTE:

This reduce function takes two arguments: a list of keys and a list of values. For our summing purposes we can ignore the keys-list and consider only the value list. We’re looping over the list and add each item to a running total that we’re returning at the end of the function.

You’ll see one difference between the map and the reduce function. The map function uses emit() to create its result, whereas the reduce function returns a value.

For example, from a list of integer values that specify the age, calculate the sum of all years of life for the news headline, “786 life years present at event.” A little contrived, but very simple and thus good for demonstration purposes. Consider the documents and the map view we used earlier in this document.

The reduce function to calculate the total age of all girls is:

function(keys, values) {
    return sum(values);
}

Note that, instead of the two earlier versions, we use CouchDB’s predefined sum() function. It does the same thing as the other two, but it is such a common piece of code that CouchDB has it included.

The result for our reduce view now looks like this:

{"rows":[
    {"key":null,"value":15}
]}

The total sum of all age fields in all our documents is 15. Just what we wanted. The key member of the result object is null, as we can’t know anymore which documents took part in the creation of the reduced result. We’ll cover more advanced reduce cases later on.

As a rule of thumb, the reduce function should reduce to a single scalar value. That is, an integer; a string; or a small, fixed-size list or object that includes an aggregated value (or values) from the values argument. It should never just return values or similar. CouchDB will give you a warning if you try to use reduce “the wrong way”:

{
    "error":"reduce_overflow_error",
    "message":"Reduce output must shrink more rapidly: Current output: ..."
}

Get Unique Values

SELECT DISTINCT field FROM table

How you can do this in CouchDB?

Getting unique values is not as easy as adding a keyword. But a reduce view and a special query parameter give us the same result. Let’s say you want a list of tags that your users have tagged themselves with and no duplicates.

First, let’s look at the source documents. We punt on _id and _rev attributes here:

{
    "name":"Chris",
    "tags":["mustache", "music", "couchdb"]
}

{
    "name":"Noah",
    "tags":["hypertext", "philosophy", "couchdb"]
}

{
    "name":"Jan",
    "tags":["drums", "bike", "couchdb"]
}

Next, we need a list of all tags. A map function will do the trick:

function(doc) {
    if(doc.name && doc.tags) {
        doc.tags.forEach(function(tag) {
            emit(tag, null);
        });
    }
}

The result will look like this:

{"total_rows":9,"offset":0,"rows":[
{"id":"3525ab874bc4965fa3cda7c549e92d30","key":"bike","value":null},
{"id":"3525ab874bc4965fa3cda7c549e92d30","key":"couchdb","value":null},
{"id":"53f82b1f0ff49a08ac79a9dff41d7860","key":"couchdb","value":null},
{"id":"da5ea89448a4506925823f4d985aabbd","key":"couchdb","value":null},
{"id":"3525ab874bc4965fa3cda7c549e92d30","key":"drums","value":null},
{"id":"53f82b1f0ff49a08ac79a9dff41d7860","key":"hypertext","value":null},
{"id":"da5ea89448a4506925823f4d985aabbd","key":"music","value":null},
{"id":"da5ea89448a4506925823f4d985aabbd","key":"mustache","value":null},
{"id":"53f82b1f0ff49a08ac79a9dff41d7860","key":"philosophy","value":null}
]}

As promised, these are all the tags, including duplicates. Since each document gets run through the map function in isolation, it cannot know if the same key has been emitted already. At this stage, we need to live with that. To achieve uniqueness, we need a reduce:

function(keys, values) {
    return true;
}

This reduce doesn’t do anything, but it allows us to specify a special query parameter when querying the view:

/dudes/_design/dude-data/_view/tags?group=true

CouchDB replies:

{"rows":[
{"key":"bike","value":true},
{"key":"couchdb","value":true},
{"key":"drums","value":true},
{"key":"hypertext","value":true},
{"key":"music","value":true},
{"key":"mustache","value":true},
{"key":"philosophy","value":true}
]}

In this case, we can ignore the value part because it is always true, but the result includes a list of all our tags and no duplicates!

With a small change we can put the reduce to good use, too. Let’s see how many of the non-unique tags are there for each tag. To calculate the tag frequency, we just use the summing up we already learned about. In the map function, we emit a 1 instead of null:

function(doc) {
    if(doc.name && doc.tags) {
        doc.tags.forEach(function(tag) {
            emit(tag, 1);
        });
    }
}

In the reduce function, we return the sum of all values:

function(keys, values) {
    return sum(values);
}

Now, if we query the view with the ?group=true parameter, we get back the count for each tag:

{"rows":[
{"key":"bike","value":1},
{"key":"couchdb","value":3},
{"key":"drums","value":1},
{"key":"hypertext","value":1},
{"key":"music","value":1},
{"key":"mustache","value":1},
{"key":"philosophy","value":1}
]}

Enforcing Uniqueness

UNIQUE KEY(column)

How you can do this in CouchDB?

Use case: your applications require that a certain value exists only once in a database.

This is an easy one: within a CouchDB database, each document must have a unique _id field. If you require unique values in a database, just assign them to a document’s _id field and CouchDB will enforce uniqueness for you.

There’s one caveat, though: in the distributed case, when you are running more than one CouchDB node that accepts write requests, uniqueness can be guaranteed only per node or outside of CouchDB. CouchDB will allow two identical IDs to be written to two different nodes. On replication, CouchDB will detect a conflict and flag the document accordingly.

Pagination Recipe

We assume you’re familiar with creating and querying documents and views as well as the multiple view query options.

Example Data

{ "name":"Biffy Clyro" }
{ "name":"Foo Fighters" }
{ "name":"Tool" }
{ "name":"Nirvana" }
{ "name":"Helmet" }
{ "name":"Tenacious D" }
{ "name":"Future of the Left" }
{ "name":"A Perfect Circle" }
{ "name":"Silverchair" }
{ "name":"Queens of the Stone Age" }
{ "name":"Kerub" }

A View

function(doc) {
    if(doc.name) {
        var name = doc.name.replace(/^(A|The) /, "");
        emit(name, null);
    }
}

The views result is an alphabetical list of band names. Now say we want to display band names five at a time and have a link pointing to the next five names that make up one page, and a link for the previous five, if we’re not on the first page.

We learned how to use the startkey, limit, and skip parameters in earlier documents. We’ll use these again here. First, let’s have a look at the full result set:

{"total_rows":11,"offset":0,"rows":[
    {"id":"a0746072bba60a62b01209f467ca4fe2","key":"Biffy Clyro","value":null},
    {"id":"b47d82284969f10cd1b6ea460ad62d00","key":"Foo Fighters","value":null},
    {"id":"45ccde324611f86ad4932555dea7fce0","key":"Tenacious D","value":null},
    {"id":"d7ab24bb3489a9010c7d1a2087a4a9e4","key":"Future of the Left","value":null},
    {"id":"ad2f85ef87f5a9a65db5b3a75a03cd82","key":"Helmet","value":null},
    {"id":"a2f31cfa68118a6ae9d35444fcb1a3cf","key":"Nirvana","value":null},
    {"id":"67373171d0f626b811bdc34e92e77901","key":"Kerub","value":null},
    {"id":"3e1b84630c384f6aef1a5c50a81e4a34","key":"Perfect Circle","value":null},
    {"id":"84a371a7b8414237fad1b6aaf68cd16a","key":"Queens of the Stone Age","value":null},
    {"id":"dcdaf08242a4be7da1a36e25f4f0b022","key":"Silverchair","value":null},
    {"id":"fd590d4ad53771db47b0406054f02243","key":"Tool","value":null}
]}

Setup

Or in a pseudo-JavaScript snippet:

var result = new Result();
var page = result.getPage();
page.display();
if(result.hasPrev()) {
    page.display_link('prev');
}
if(result.hasNext()) {
    page.display_link('next');
}

Paging

curl -X GET http://127.0.0.1:5984/artists/_design/artists/_view/by-name?limit=5

The result:

{"total_rows":11,"offset":0,"rows":[
    {"id":"a0746072bba60a62b01209f467ca4fe2","key":"Biffy Clyro","value":null},
    {"id":"b47d82284969f10cd1b6ea460ad62d00","key":"Foo Fighters","value":null},
    {"id":"45ccde324611f86ad4932555dea7fce0","key":"Tenacious D","value":null},
    {"id":"d7ab24bb3489a9010c7d1a2087a4a9e4","key":"Future of the Left","value":null},
    {"id":"ad2f85ef87f5a9a65db5b3a75a03cd82","key":"Helmet","value":null}
]}

By comparing the total_rows value to our limit value, we can determine if there are more pages to display. We also know by the offset member that we are on the first page. We can calculate the value for skip= to get the results for the next page:

var rows_per_page = 5;
var page = (offset / rows_per_page) + 1; // == 1
var skip = page * rows_per_page; // == 5 for the first page, 10 for the second ...

So we query CouchDB with:

curl -X GET 'http://127.0.0.1:5984/artists/_design/artists/_view/by-name?limit=5&skip=5'

Note we have to use ' (single quotes) to escape the & character that is special to the shell we execute curl in.

The result:

{"total_rows":11,"offset":5,"rows":[
    {"id":"a2f31cfa68118a6ae9d35444fcb1a3cf","key":"Nirvana","value":null},
    {"id":"67373171d0f626b811bdc34e92e77901","key":"Kerub","value":null},
    {"id":"3e1b84630c384f6aef1a5c50a81e4a34","key":"Perfect Circle","value":null},
    {"id":"84a371a7b8414237fad1b6aaf68cd16a","key":"Queens of the Stone Age",
    "value":null},
    {"id":"dcdaf08242a4be7da1a36e25f4f0b022","key":"Silverchair","value":null}
]}

Implementing the hasPrev() and hasNext() method is pretty straightforward:

function hasPrev()
{
    return page > 1;
}
function hasNext()
{
    var last_page = Math.floor(total_rows / rows_per_page) +
        (total_rows % rows_per_page);
    return page != last_page;
}

Paging (Alternate Method)

A correct solution is not much harder. Instead of slicing the result set into equally sized pages, we look at 10 rows at a time and use startkey to jump to the next 10 rows. We even use skip, but only with the value 1.

Here is how it works:

The trick to finding the next page is pretty simple. Instead of requesting 10 rows for a page, you request 11 rows, but display only 10 and use the values in the 11th row as the startkey for the next page. Populating the link to the previous page is as simple as carrying the current startkey over to the next page. If there’s no previous startkey, we are on the first page. We stop displaying the link to the next page if we get rows_per_page or less rows back. This is called linked list pagination, as we go from page to page, or list item to list item, instead of jumping directly to a pre-computed page. There is one caveat, though. Can you spot it?

CouchDB view keys do not have to be unique; you can have multiple index entries read. What if you have more index entries for a key than rows that should be on a page? startkey jumps to the first row, and you’d be screwed if CouchDB didn’t have an additional parameter for you to use. All view keys with the same value are internally sorted by docid, that is, the ID of the document that created that view row. You can use the startkey_docid and endkey_docid parameters to get subsets of these rows. For pagination, we still don’t need endkey_docid, but startkey_docid is very handy. In addition to startkey and limit, you also use startkey_docid for pagination if, and only if, the extra row you fetch to find the next page has the same key as the current startkey.

It is important to note that the *_docid parameters only work in addition to the *key parameters and are only useful to further narrow down the result set of a view for a single key. They do not work on their own (the one exception being the built-in _all_docs view that already sorts by document ID).

The advantage of this approach is that all the key operations can be performed on the super-fast B-tree index behind the view. Looking up a page doesn’t include scanning through hundreds and thousands of rows unnecessarily.

Jump to Page

If you really do need to jump to a page over the full range of documents (we have seen applications that require that), you can still maintain an integer value index as the view index and take a hybrid approach at solving pagination.

Search

WARNING:

To create a search index, you add a JavaScript function to a design document in the database. An index builds after processing one search request or after the server detects a document update. The index function takes the following parameters:

1. Field name - The name of the field you want to use when you query the index. If you set this parameter to default, then this field is queried if no field is specified in the query syntax.

3. (Optional) The third parameter includes the following fields: boost, facet, index, and store. These fields are described in more detail later.

By default, a search index response returns 25 rows. The number of rows that is returned can be changed by using the limit parameter. Each response includes a bookmark field. You can include the value of the bookmark field in later queries to look through the responses.

Example design document that defines a search index:

{
    "_id": "_design/search_example",
    "indexes": {
        "animals": {
            "index": "function(doc){ ... }"
        }
    }
}

A search index will inherit the partitioning type from the options.partitioned field of the design document that contains it.

Index functions

NOTE:

NOTE:

index("myfield", "this is a string");
index("myfield", 123);

The function that is contained in the index field is a JavaScript function that is called for each document in the database. The function takes the document as a parameter, extracts some data from it, and then calls the function that is defined in the index field to index that data.

The index function takes three parameters, where the third parameter is optional.

The first parameter is the name of the field you intend to use when querying the index, and which is specified in the Lucene syntax portion of subsequent queries. An example appears in the following query:

query=color:red

The Lucene field name color is the first parameter of the index function.

The query parameter can be abbreviated to q, so another way of writing the query is as follows:

q=color:red

If the special value "default" is used when you define the name, you do not have to specify a field name at query time. The effect is that the query can be simplified:

query=red

The second parameter is the data to be indexed. Keep the following information in mind when you index your data:

The third, optional, parameter is a JavaScript object with the following fields:

Index function (optional parameter)

NOTE:

Example search index function:

function(doc) {
    index("default", doc._id);
    if (doc.min_length) {
        index("min_length", doc.min_length, {"store": true});
    }
    if (doc.diet) {
        index("diet", doc.diet, {"store": true});
    }
    if (doc.latin_name) {
        index("latin_name", doc.latin_name, {"store": true});
    }
    if (doc.class) {
        index("class", doc.class, {"store": true});
    }
}

Index guard clauses

Example of failing to check whether the index data field exists:

if (doc.min_length) {
    index("min_length", doc.min_length, {"store": true});
}

You might use the JavaScript typeof function to implement the guard clause test. If the field exists and has the expected type, the correct type name is returned, so the guard clause test succeeds and it is safe to use the index function. If the field does not exist, you would not get back the expected type of the field, therefore you would not attempt to index the field.

JavaScript considers a result to be false if one of the following values is tested:

Using a guard clause to check whether the required data field exists, and holds a number, before an attempt to index:

if (typeof(doc.min_length) === 'number') {
    index("min_length", doc.min_length, {"store": true});
}

Use a generic guard clause test to ensure that the type of the candidate data field is defined.

Example of a ‘generic’ guard clause:

if (typeof(doc.min_length) !== 'undefined') {
    // The field exists, and does have a type, so we can proceed to index using it.
    ...
}

Analyzers

Here’s the list of generic analyzers, and their descriptions, that are supported by search:

Example analyzer document:

{
    "_id": "_design/analyzer_example",
    "indexes": {
        "INDEX_NAME": {
            "index": "function (doc) { ... }",
            "analyzer": "$ANALYZER_NAME"
        }
    }
}

Language-specific analyzers

NOTE:

NOTE:

Per-field analyzers

Example of defining different analyzers for different fields:

{
    "_id": "_design/analyzer_example",
    "indexes": {
        "INDEX_NAME": {
            "analyzer": {
                "name": "perfield",
                "default": "english",
                "fields": {
                    "spanish": "spanish",
                    "german": "german"
                }
            },
            "index": "function (doc) { ... }"
        }
    }
}

Stop words

NOTE:

The default stop words for the standard analyzer are included below:

"a", "an", "and", "are", "as", "at", "be", "but", "by", "for", "if",
"in", "into", "is", "it", "no", "not", "of", "on", "or", "such",
"that", "the", "their", "then", "there", "these", "they", "this",
"to", "was", "will", "with"

Example of defining non-indexed (‘stop’) words:

{
    "_id": "_design/stop_words_example",
    "indexes": {
        "INDEX_NAME": {
            "analyzer": {
                "name": "portuguese",
                "stopwords": [
                    "foo",
                    "bar",
                    "baz"
                ]
            },
            "index": "function (doc) { ... }"
        }
    }
}

Testing analyzer tokenization

Example of using HTTP to test the keyword analyzer:

POST /_search_analyze HTTP/1.1
Content-Type: application/json
{"analyzer":"keyword", "text":"ablanks@renovations.com"}

Example of using the command line to test the keyword analyzer:

curl 'https://$HOST:5984/_search_analyze' -H 'Content-Type: application/json'
    -d '{"analyzer":"keyword", "text":"ablanks@renovations.com"}'

Result of testing the keyword analyzer:

{
    "tokens": [
        "ablanks@renovations.com"
    ]
}

Example of using HTTP to test the standard analyzer:

POST /_search_analyze HTTP/1.1
Content-Type: application/json
{"analyzer":"standard", "text":"ablanks@renovations.com"}

Example of using the command line to test the standard analyzer:

curl 'https://$HOST:5984/_search_analyze' -H 'Content-Type: application/json'
    -d '{"analyzer":"standard", "text":"ablanks@renovations.com"}'

Result of testing the standard analyzer:

{
    "tokens": [
        "ablanks",
        "renovations.com"
    ]
}

Queries

Specify your search by using the query parameter.

Example of using HTTP to query a partitioned index:

GET /$DATABASE/_partition/$PARTITION_KEY/_design/$DDOC/_search/$INDEX_NAME?include_docs=true&query="*:*"&limit=1 HTTP/1.1
Content-Type: application/json

Example of using HTTP to query a global index:

GET /$DATABASE/_design/$DDOC/_search/$INDEX_NAME?include_docs=true&query="*:*"&limit=1 HTTP/1.1
Content-Type: application/json

Example of using the command line to query a partitioned index:

curl https://$HOST:5984/$DATABASE/_partition/$PARTITION_KEY/_design/$DDOC/
_search/$INDEX_NAME?include_docs=true\&query="*:*"\&limit=1 \

Example of using the command line to query a global index:

curl https://$HOST:5984/$DATABASE/_design/$DDOC/_search/$INDEX_NAME?
include_docs=true\&query="*:*"\&limit=1 \

Query Parameters

You must enable faceting before you can use the following parameters:

NOTE:

Relevance

Relevance is measured according to Apache Lucene Scoring. As an example, if you search a simple database for the word example, two documents might contain the word. If one document mentions the word example 10 times, but the second document mentions it only twice, then the first document is considered to be more ‘relevant’.

If you do not provide a sort parameter, relevance is used by default. The highest scoring matches are returned first.

If you provide a sort parameter, then matches are returned in that order, ignoring relevance.

If you want to use a sort parameter, and also include ordering by relevance in your search results, use the special fields -<score> or <score> within the sort parameter.

POSTing search queries

Example of using HTTP to POST a search request:

POST /db/_design/ddoc/_search/searchname HTTP/1.1
Content-Type: application/json

Example of using the command line to POST a search request:

curl 'https://$HOST:5984/db/_design/ddoc/_search/searchname' -X POST -H 'Content-Type: application/json' -d @search.json

Example JSON document that contains a search request:

{
    "q": "index:my query",
    "sort": "foo",
    "limit": 3
}

Query syntax

Example search query expressions:

// Birds
class:bird

// Animals that begin with the letter "l"
l*

// Carnivorous birds
class:bird AND diet:carnivore

// Herbivores that start with letter "l"
l* AND diet:herbivore

// Medium-sized herbivores
min_length:[1 TO 3] AND diet:herbivore

// Herbivores that are 2m long or less
diet:herbivore AND min_length:[-Infinity TO 2]

// Mammals that are at least 1.5m long
class:mammal AND min_length:[1.5 TO Infinity]

// Find "Meles meles"
latin_name:"Meles meles"

// Mammals who are herbivore or carnivore
diet:(herbivore OR omnivore) AND class:mammal

// Return all results
*:*

Queries over multiple fields can be logically combined, and groups and fields can be further grouped. The available logical operators are case-sensitive and are AND, +, OR, NOT and -. Range queries can run over strings or numbers.

If you want a fuzzy search, you can run a query with ~ to find terms like the search term. For instance, look~ finds the terms book and took.

NOTE:

You can alter the importance of a search term by adding ^ and a positive number. This alteration makes matches containing the term more or less relevant, proportional to the power of the boost value. The default value is 1, which means no increase or decrease in the strength of the match. A decimal value of 0 - 1 reduces importance. making the match strength weaker. A value greater than one increases importance, making the match strength stronger.

Wildcard searches are supported, for both single (?) and multiple (*) character searches. For example, dat? would match date and data, whereas dat* would match date, data, database, and dates. Wildcards must come after the search term.

Use *:* to return all results.

If the search query does not specify the "group_field" argument, the response contains a bookmark. If this bookmark is later provided as a URL parameter, the response skips the rows that were seen already, making it quick and easy to get the next set of results.

NOTE:

NOTE:

The following characters require escaping if you want to search on them:

+ - && || ! ( ) { } [ ] ^ " ~ * ? : \ /

To escape one of these characters, use a preceding backslash character (\).

The response to a search query contains an order field for each of the results. The order field is an array where the first element is the field or fields that are specified in the sort parameter. See the sort parameter. If no sort parameter is included in the query, then the order field contains the Lucene relevance score. If you use the ‘sort by distance’ feature as described in geographical searches, then the first element is the distance from a point. The distance is measured by using either kilometers or miles.

NOTE:

Faceting

Example of search query, specifying that faceted search is enabled:

function(doc) {
    index("type", doc.type, {"facet": true});
    index("price", doc.price, {"facet": true});
}

To use facets, all the documents in the index must include all the fields that have faceting enabled. If your documents do not include all the fields, you receive a bad_request error with the following reason, “The field_name does not exist.” If each document does not contain all the fields for facets, create separate indexes for each field. If you do not create separate indexes for each field, you must include only documents that contain all the fields. Verify that the fields exist in each document by using a single if statement.

Example if statement to verify that the required fields exist in each document:

if (typeof doc.town == "string" && typeof doc.name == "string") {
    index("town", doc.town, {facet: true});
    index("name", doc.name, {facet: true});
   }

Counts

The counts facet syntax takes a list of fields, and returns the number of query results for each unique value of each named field.

NOTE:

Example of a query using the counts facet syntax:

?q=*:*&counts=["type"]

Example response after using of the counts facet syntax:

{
    "total_rows":100000,
    "bookmark":"g...",
    "rows":[...],
    "counts":{
        "type":{
            "sofa": 10,
            "chair": 100,
            "lamp": 97
        }
    }
}

Drilldown

You can restrict results to documents with a dimension equal to the specified label. Restrict the results by adding drilldown=["dimension","label"] to a search query. You can include multiple drilldown parameters to restrict results along multiple dimensions.

GET /things/_design/inventory/_search/fruits?q=*:*&drilldown=["state","old"]&drilldown=["item","apple"]&include_docs=true HTTP/1.1

For better language interoperability, you can achieve the same by supplying a list of lists:

GET /things/_design/inventory/_search/fruits?q=*:*&drilldown=[["state","old"],["item","apple"]]&include_docs=true HTTP/1.1

You can also supply a list of lists for drilldown in bodies of POST requests.

Note that, multiple values for a single key in a drilldown means an OR relation between them and there is an AND relation between multiple keys.

Using a drilldown parameter is similar to using key:value in the q parameter, but the drilldown parameter returns values that the analyzer might skip.

For example, if the analyzer did not index a stop word like "a", using drilldown returns it when you specify drilldown=["key","a"].

Ranges

The range facet syntax reuses the standard Lucene syntax for ranges to return counts of results that fit into each specified category. Inclusive range queries are denoted by brackets ([, ]). Exclusive range queries are denoted by curly brackets ({, }).

NOTE:

Example of a request that uses faceted search for matching ranges:

?q=*:*&ranges={"price":{"cheap":"[0 TO 100]","expensive":"{100 TO Infinity}"}}

Example results after a ranges check on a faceted search:

{
    "total_rows":100000,
    "bookmark":"g...",
    "rows":[...],
    "ranges": {
        "price": {
            "expensive": 278682,
            "cheap": 257023
        }
    }
}

Geographical searches

To sort your results in this way, you must index two numeric fields, representing the longitude and latitude.

NOTE:

You can then query by using the special <distance...> sort field, which takes five parameters:

You can combine sorting by distance with any other search query, such as range searches on the latitude and longitude, or queries that involve non-geographical information.

That way, you can search in a bounding box, and narrow down the search with extra criteria.

Example geographical data:

{
    "name":"Aberdeen, Scotland",
    "lat":57.15,
    "lon":-2.15,
    "type":"city"
}

Example of a design document that contains a search index for the geographic data:

function(doc) {
    if (doc.type && doc.type == 'city') {
        index('city', doc.name, {'store': true});
        index('lat', doc.lat, {'store': true});
        index('lon', doc.lon, {'store': true});
    }
}

An example of using HTTP for a query that sorts cities in the northern hemisphere by their distance to New York:

GET /examples/_design/cities-designdoc/_search/cities?q=lat:[0+TO+90]&sort="<distance,lon,lat,-74.0059,40.7127,km>" HTTP/1.1

An example of using the command line for a query that sorts cities in the northern hemisphere by their distance to New York:

curl 'https://$HOST:5984/examples/_design/cities-designdoc/_search/cities?q=lat:[0+TO+90]&sort="<distance,lon,lat,-74.0059,40.7127,km>"'

Example (abbreviated) response, containing a list of northern hemisphere cities sorted by distance to New York:

{
    "total_rows": 205,
    "bookmark": "g1A...XIU",
    "rows": [
        {
            "id": "city180",
            "order": [
                8.530665755719783,
                18
            ],
            "fields": {
                "city": "New York, N.Y.",
                "lat": 40.78333333333333,
                "lon": -73.96666666666667
            }
        },
        {
            "id": "city177",
            "order": [
                13.756343205985946,
                17
            ],
            "fields": {
                "city": "Newark, N.J.",
                "lat": 40.733333333333334,
                "lon": -74.16666666666667
            }
        },
        {
            "id": "city178",
            "order": [
                113.53603438866077,
                26
            ],
            "fields": {
                "city": "New Haven, Conn.",
                "lat": 41.31666666666667,
                "lon": -72.91666666666667
            }
        }
    ]
}

Highlighting search terms

To get more emphasized results, add the highlight_fields parameter to the search query. Specify the field names for which you would like excerpts, with the highlighted search term returned.

By default, the search term is placed in <em> tags to highlight it, but the highlight can be overridden by using the highlights_pre_tag and highlights_post_tag parameters.

The length of the fragments is 100 characters by default. A different length can be requested with the highlights_size parameter.

The highlights_number parameter controls the number of fragments that are returned, and defaults to 1.

In the response, a highlights field is added, with one subfield per field name.

For each field, you receive an array of fragments with the search term highlighted.

NOTE:

Example of using HTTP to search with highlighting enabled:

GET /movies/_design/searches/_search/movies?q=movie_name:Azazel&highlight_fields=["movie_name"]&highlight_pre_tag="**"&highlight_post_tag="**"&highlights_size=30&highlights_number=2 HTTP/1.1
Authorization: ...

Example of using the command line to search with highlighting enabled:

curl "https://$HOST:5984/movies/_design/searches/_search/movies?q=movie_name:Azazel&highlight_fields=\[\"movie_name\"\]&highlight_pre_tag=\"**\"&highlight_post_tag=\"**\"&highlights_size=30&highlights_number=2

Example of highlighted search results:

{
    "highlights": {
        "movie_name": [
            " on the Azazel Orient Express",
            " Azazel manuals, you"
        ]
    }
}

Note: Previously, the functionality provided by CouchDB’s design documents, in combination with document attachments, was referred to as “CouchApps.” The general principle was that entire web applications could be hosted in CouchDB, without need for an additional application server.

Use of CouchDB as a combined standalone database and application server is no longer recommended. There are significant limitations to a pure CouchDB web server application stack, including but not limited to: fully-fledged fine-grained security, robust templating and scaffolding, complete developer tooling, and most importantly, a thriving ecosystem of developers, modules and frameworks to choose from.

The developers of CouchDB believe that web developers should pick “the right tool for the right job”. Use CouchDB as your database layer, in conjunction with any number of other server-side web application frameworks, such as the entire Node.JS ecosystem, Python’s Django and Flask, PHP’s Drupal, Java’s Apache Struts, and more.

Document Design Considerations

Don’t rely on CouchDB’s auto-UUID generation

Alternatives to auto-incrementing sequences

Pre-aggregating your data

In this case, using an in-memory store to summarize your statistical information, then writing out to CouchDB every 10 seconds / 1 minute / whatever level of granularity you need would greatly reduce the number of documents you’ll put in your database.

Later, you can then further decimate your data by walking the entire database and generating documents to be stored in a new database with a lower level of granularity (say, 1 document a day). You can then delete the older, more fine-grained database when you’re done with it.

Designing an application to work with replication

Here we consider a simple example of a bookmarks application. The idea is that a user can replicate their own bookmarks, work with them on another machine, and then synchronise their changes later.

Let’s start with a very simple definition of bookmarks: an ordered, nestable mapping of name to URL. Internally the application might represent it like this:

[
  {"name":"Weather", "url":"http://www.bbc.co.uk/weather"},
  {"name":"News", "url":"http://news.bbc.co.uk/"},
  {"name":"Tech", "bookmarks": [
    {"name":"Register", "url":"http://www.theregister.co.uk/"},
    {"name":"CouchDB", "url":"http://couchdb.apache.org/"}
  ]}
]

It can then present the bookmarks menu and sub-menus by traversing this structure.

Now consider this scenario: the user has a set of bookmarks on her PC, and then replicates it to her laptop. On the laptop, she changes the News link to point to CNN, renames “Register” to “The Register”, and adds a new link to slashdot just after it. On the desktop, her husband deletes the Weather link, and adds a new link to CNET in the Tech folder.

So after these changes, the laptop has:

[
  {"name":"Weather", "url":"http://www.bbc.co.uk/weather"},
  {"name":"News", "url":"http://www.cnn.com/"},
  {"name":"Tech", "bookmarks": [
    {"name":"The Register", "url":"http://www.theregister.co.uk/"},
    {"name":"Slashdot", "url":"http://www.slashdot.new/"},
    {"name":"CouchDB", "url":"http://couchdb.apache.org/"}
  ]}
]

and the PC has:

[
  {"name":"News", "url":"http://www.cnn.com/"},
  {"name":"Tech", "bookmarks": [
    {"name":"Register", "url":"http://www.theregister.co.uk/"},
    {"name":"CouchDB", "url":"http://couchdb.apache.org/"},
    {"name":"CNET", "url":"http://news.cnet.com/"}
  ]}
]

Upon the next synchronisation, we want the expected merge to take place. That is: links which were changed, added or deleted on one side are also changed, added or deleted on the other side - with no human intervention required unless absolutely necessary.

We will also assume that both sides are doing a CouchDB “compact” operation periodically, and are disconnected for more than this time before they resynchronise.

All of the approaches below which allow automated merging of changes rely on having some sort of history, back to the point where the replicas diverged.

CouchDB does not provide a mechanism for this itself. It stores arbitrary numbers of old _ids for one document (trunk now has a mechanism for pruning the _id history), for the purposes of replication. However it will not keep the documents themselves through a compaction cycle, except where there are conflicting versions of a document.

Do not rely on the CouchDB revision history mechanism to help you build an application-level version history. Its sole purpose is to ensure eventually consistent replication between databases. It is up to you to maintain history explicitly in whatever form makes sense for your application, and to prune it to avoid excessive storage utilisation, whilst not pruning past the point where live replicas last diverged.

Approach 1: Single JSON doc

{
  "bookmarks":
  // ... same as above
}

This makes life very easy for the application, as the ordering and nesting is all taken care of. The trouble here is that on replication, only two sets of bookmarks will be visible: example B and example C. One will be chosen as the main revision, and the other will be stored as a conflicting revision.

At this point, the semantics are very unsatisfactory from the user’s point of view. The best that can be offered is a choice saying “Which of these two sets of bookmarks do you wish to keep: B or C?” However neither represents the desired outcome. There is also insufficient data to be able to correctly merge them, since the base revision A is lost.

This is going to be highly unsatisfactory for the user, who will have to apply one set of changes again manually.

Approach 2: Separate document per bookmark

Since there will now be lots of small documents, you may either wish to keep a completely separate database for bookmarks, or else add an attribute to distinguish bookmarks from other kinds of document in the database. In the latter case, a view can be made to return only bookmark documents.

Whilst replication is now fixed, care is needed with the “ordered” and “nestable” properties of bookmarks.

For ordering, one suggestion is to give each item a floating-point index, and then when inserting an object between A and B, give it an index which is the average of A and B’s indices. Unfortunately, this will fail after a while when you run out of precision, and the user will be bemused to find that their most recent bookmarks no longer remember the exact position they were put in.

A better way is to keep a string representation of index, which can grow as the tree is subdivided. This will not suffer the above problem, but it may result in this string becoming arbitrarily long after time. They could be renumbered, but the renumbering operation could introduce a lot of conflicts, especially if attempted by both sides independently.

For “nestable”, you can have a separate doc which represents a list of bookmarks, and each bookmark can have a “belongs to” field which identifies the list. It may be useful anyway to be able to have multiple top-level bookmark sets (Bob’s bookmarks, Jill’s bookmarks etc). Some care is needed when deleting a list or sub-list, to ensure that all associated bookmarks are also deleted, otherwise they will become orphaned.

Building the entire bookmark set can be performed through the use of emitting a compound key that describes the path to the document, then using group levels to retrieve the position of the tree in the document. The following code excerpt describes a tree of files, where the path to the file is stored in the document under the "path" key:

// map function
function(doc) {
  if (doc.type === "file") {
    if (doc.path.substr(-1) === "/") {
      var raw_path = doc.path.slice(0, -1);
    } else {
      var raw_path = doc.path;
    }
    emit (raw_path.split('/'), 1);
  }
}
// reduce
_sum

This will emit rows into the view of the form ["opt", "couchdb", "etc", "local.ini"] for a doc.path of /opt/couchdb/etc/local.ini. You can then query a list of files in the /opt/couchdb/etc directory by specifying a startkey of ["opt", "couchdb", "etc"] and an endkey of ["opt", "couchdb", "etc", {}].

Approach 3: Immutable history / event sourcing

In this model, instead of storing individual bookmarks, you store records of changes made - “Bookmark added”, “Bookmark changed”, “Bookmark moved”, “Bookmark deleted”. These are stored in an append-only fashion. Since records are never modified or deleted, only added to, there are never any replication conflicts.

These records can also be stored as an array in a single CouchDB document. Replication can cause a conflict, but in this case it is easy to resolve by simply combining elements from the two arrays.

In order to see the full set of bookmarks, you need to start with a baseline set (initially empty) and run all the change records since the baseline was created; and/or you need to maintain a most-recent version and update it with changes not yet seen.

Care is needed after replication when merging together history from multiple sources. You may get different results depending on how you order them - consider taking all A’s changes before B’s, taking all B’s before A’s, or interleaving them (e.g. if each change has a timestamp).

Also, over time the amount of storage used can grow arbitrarily large, even if the set of bookmarks itself is small. This can be controlled by moving the baseline version forwards and then keeping only the changes after that point. However, care is needed not to move the baseline version forward so far that there are active replicas out there which last synchronised before that time, as this may result in conflicts which cannot be resolved automatically.

If there is any uncertainty, it is best to present the user with a prompt to assist with merging the content in the application itself.

Approach 4: Keep historic versions explicitly

You can keep a pointer to the ‘most current’ revision, and each revision can point to its predecessor. On replication, merging can take place by diffing each of the previous versions (in effect synthesising the command logs) back to a common ancestor.

This is the sort of behaviour which revision control systems such as Git implement as a matter of routine, although generally comparing text files line-by-line rather than comparing JSON objects field-by-field.

Systems like Git will accumulate arbitrarily large amounts of history (although they will attempt to compress it by packing multiple revisions so that only their diffs are stored). With Git you can use “history rewriting” to remove old history, but this may prohibit merging if history doesn’t go back far enough in time.

Adding client-side security with a translucent database

The simplest solutions use a one-way function like SHA-256 at the client to scramble the name and password before storing the information. This solution gives the client control of the data in the database without requiring a thick layer on the database to test each transaction. Some advantages are:

There are limitations:

There are many variations on this theme detailed in the book Translucent Databases, including:

Document submission using HTML Forms

The HTML form

<form action="/dbname/_design/ddocname/_update/contactform" method="post">
    <div>
        <label for="name">Name:</label>
        <input type="text" id="name" name="name" />
    </div>
    <div>
        <label for="mail">Email:</label>
        <input type="text" id="mail" name="email" />
    </div>
    <div>
        <label for="msg">Message:</label>
        <textarea id="msg" name="message"></textarea>
    </div>
</form>

Customize the /dbname/_design/ddocname/_update/contactform portion of the form action URL to reflect the exact path to your database, design document and update function (see below).

As CouchDB no longer recommends the use of CouchDB-hosted web applications , you may want to use a reverse proxy to expose CouchDB as a subdirectory of your web application. If so, add that prefix to the action destination in the form.

Another option is to alter CouchDB’s CORS settings and use a cross-domain POST. Be sure you understand all security implications before doing this!

The update function

The first argument to the function will be the document that is being processed (if it exists). Because we are using POST and not PUT, this should be empty in our scenario - but we should check to be sure. The POST-ed data will be passed as the second parameter to the function, along with any query parameters and the full request headers.

Here’s a sample handler that extracts the form data, generates a document _id based on the email address and timestamp, and saves the document. It then returns a JSON success response back to the browser.

function(doc, req) {
    if (doc) {
        return [doc, toJSON({"error": "request already filed"})]
    }
    if !(req.form && req.form.email) {
        return [null, toJSON({"error": "incomplete form"})]
    }
    var date = new Date()
    var newdoc = req.form
    newdoc._id = req.form.email + "_" + date.toISOString()
    return [newdoc, toJSON({"success":"ok"})]
}

Place the above function in your design document under the updates key.

Note that this function does not attempt any sort of input validation or sanitization. That is best handled by a validate document update function instead. (A “VDU” will validate any document written to the database, not just those that use your update function.)

If the first element passed to return is a document, the HTTP response headers will include X-Couch-Id, the _id value for the newly created document, and X-Couch-Update-NewRev, the _rev value for the newly created document. This is handy if your client-side code wants to access or update the document in a future call.

Example output

$ curl -X PUT localhost:5984/testdb/_design/myddoc -d '{ "updates": { "contactform": "function(doc, req) { ... }" } }'
{"ok":true,"id":"_design/myddoc","rev":"1-2a2b0951fcaf7287817573b03bba02ed"}
$ curl --data "name=Lin&email=lin@example.com&message=I Love CouchDB" http://localhost:5984/testdb/_design/myddoc/_update/contactform
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5984 (#1)
> POST /testdb/_design/myddoc/_update/contactform HTTP/1.1
> Host: localhost:5984
> User-Agent: curl/7.59.0
> Accept: */*
> Content-Length: 53
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 53 out of 53 bytes
< HTTP/1.1 201 Created
< Content-Length: 16
< Content-Type: text/html; charset=utf-8
< Date: Thu, 05 Apr 2018 19:56:42 GMT
< Server: CouchDB/2.2.0-948a1311c (Erlang OTP/19)
< X-Couch-Id: lin%40example.com_2018-04-05T19:51:22.278Z
< X-Couch-Request-ID: 03a5f4fbe0
< X-Couch-Update-NewRev: 1-34483732407fcc6cfc5b60ace48b9da9
< X-CouchDB-Body-Time: 0
<
* Connection #1 to host localhost left intact
{"success":"ok"}
$ curl http://localhost:5984/testdb/lin\@example.com_2018-04-05T19:51:22.278Z
{"_id":"lin@example.com_2018-04-05T19:51:22.278Z","_rev":"1-34483732407fcc6cfc5b60ace48b9da9","name":"Lin","email":"lin@example.com","message":"I Love CouchDB"}

Using an ISO Formatted Date for Document IDs

function(doc) {
  var dt = new Date(doc._id);
  emit([dt.getDate(), doc.widget], 1);
}

simply use group_level to zoom in on whatever time you wish to use.

curl -X GET "http://localhost:5984/transactions/_design/widget_count/_view/toss?group_level=1"
{"rows":[
{"key":[20],"value":10},
{"key":[21],"value":20}
]}
curl -X GET "http://localhost:5984/transactions/_design/widget_count/_view/toss?group_level=2"
{"rows":[
{"key":[20,widget],"value":10},
{"key":[21,widget],"value":10},
{"key":[21,thing],"value":10}
]}

Another method is using parseint() and datetime.substr() to cut out useful values for a return key:

function (doc) {
  var datetime = doc._id;
  var year = parseInt(datetime.substr(0, 4));
  var month = parseInt(datetime.substr(5, 2), 10);
  var day = parseInt(datetime.substr(8, 2), 10);
  var hour = parseInt(datetime.substr(11, 2), 10);
  var minute = parseInt(datetime.substr(14, 2), 10);
  emit([doc.widget, year, month, day, hour, minute], 1);
}

JavaScript development tips

View recommendations

Deploying a view change in a live environment

Here is a worked example, assuming your /db/_design/ddoc needs tobe updated.

The COPY HTTP verb can be used to copy the design document with a single command:

curl -X COPY <URL of source design document> -H "Destination: <ID of destination design document>"

Reverse Proxies

Reverse proxying with HAProxy

CouchDB’s sample haproxy configuration is present in the code repository and release tarball as rel/haproxy.cfg. It is included below. This example is for a 3 node CouchDB cluster:

global
    maxconn 512
    spread-checks 5
defaults
    mode http
    log global
    monitor-uri /_haproxy_health_check
    option log-health-checks
    option httplog
    balance roundrobin
    option forwardfor
    option redispatch
    retries 4
    option http-server-close
    timeout client 150000
    timeout server 3600000
    timeout connect 500
    stats enable
    stats uri /_haproxy_stats
    # stats auth admin:admin # Uncomment for basic auth
frontend http-in
     # This requires HAProxy 1.5.x
     # bind *:$HAPROXY_PORT
     bind *:5984
     default_backend couchdbs
backend couchdbs
    option httpchk GET /_up
    http-check disable-on-404
    server couchdb1 x.x.x.x:5984 check inter 5s
    server couchdb2 x.x.x.x:5984 check inter 5s
    server couchdb2 x.x.x.x:5984 check inter 5s

Reverse proxying with nginx

Basic Configuration

location / {
    proxy_pass http://localhost:5984;
    proxy_redirect off;
    proxy_buffering off;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

Proxy buffering must be disabled, or continuous replication will not function correctly behind nginx.

Reverse proxying CouchDB in a subdirectory with nginx

location /couchdb {
    rewrite /couchdb/(.*) /$1 break;
    proxy_pass http://localhost:5984;
    proxy_redirect off;
    proxy_buffering off;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

Session based replication is default functionality since CouchDB 2.3.0. To enable session based replication with reverse proxied CouchDB in a subdirectory.

location /_session {
    proxy_pass http://localhost:5984/_session;
    proxy_redirect off;
    proxy_buffering off;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

Authentication with nginx as a reverse proxy

location /couchdb {
    auth_basic "Restricted";
    auth_basic_user_file htpasswd;
    rewrite /couchdb/(.*) /$1 break;
    proxy_pass http://localhost:5984;
    proxy_redirect off;
    proxy_buffering off;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Authorization "";
}

This setup leans entirely on nginx performing authorization, and forwarding requests to CouchDB with no authentication (with CouchDB in Admin Party mode), which isn’t sufficient in CouchDB 3.0 anymore as Admin Party has been removed. You’d need to at the very least hard-code user credentials into this version with headers.

For a better solution, see api/auth/proxy.

SSL with nginx

ssl on;
ssl_certificate PATH_TO_YOUR_PUBLIC_KEY.pem;
ssl_certificate_key PATH_TO_YOUR_PRIVATE_KEY.key;
ssl_protocols SSLv3;
ssl_session_cache shared:SSL:1m;
location / {
    proxy_pass http://localhost:5984;
    proxy_redirect off;
    proxy_set_header Host $host;
    proxy_buffering off;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Ssl on;
}

The X-Forwarded-Ssl header tells CouchDB that it should use the https scheme instead of the http scheme. Otherwise, all CouchDB-generated redirects will fail.

Reverse Proxying with Caddy 2

Basic configuration

domain.com {
   reverse_proxy localhost:5984
}

Reverse proxying CouchDB in a subdirectory with Caddy 2

domain.com {
    reverse_proxy /couchdb/* localhost:5984
}

Reverse proxying + load balancing for CouchDB clusters

Caddy will check the status, i.e. health, of each node every 5 seconds; if a node goes down, Caddy will avoid proxying requests to that node until it comes back online.

domain.com {
    reverse_proxy http://localhost:15984 http://localhost:25984 http://localhost:35984 {
    lb_policy round_robin
    lb_try_interval 500ms
    health_interval 5s
    }
}

Authentication with Caddy 2 as a reverse proxy

domain.com {
    basicauth /couchdb/* {
        couch_username couchdb_hashed_password_base64
    }
    reverse_proxy /couchdb/* localhost:5984
}

This setup leans entirely on nginx performing authorization, and forwarding requests to CouchDB with no authentication (with CouchDB in Admin Party mode), which isn’t sufficient in CouchDB 3.0 anymore as Admin Party has been removed. You’d need to at the very least hard-code user credentials into this version with headers.

For a better solution, see api/auth/proxy.

Reverse Proxying with Apache HTTP Server

Basic Configuration

<VirtualHost *:80>
   ServerAdmin webmaster@dummy-host.example.com
   DocumentRoot "/opt/websites/web/www/dummy"
   ServerName couchdb.localhost
   AllowEncodedSlashes On
   ProxyRequests Off
   KeepAlive Off
   <Proxy *>
      Order deny,allow
      Deny from all
      Allow from 127.0.0.1
   </Proxy>
   ProxyPass / http://localhost:5984 nocanon
   ProxyPassReverse / http://localhost:5984
   ProxyPreserveHost On
   ErrorLog "logs/couchdb.localhost-error_log"
   CustomLog "logs/couchdb.localhost-access_log" common
</VirtualHost>

Installation on Unix-like systems

Installation using the Apache CouchDB convenience binary packages

These RedHat-style rpm packages and Debian-style deb packages will install CouchDB at /opt/couchdb and ensure CouchDB is run at system startup by the appropriate init subsystem (SysV-style initd or systemd).

The Debian-style deb packages also pre-configure CouchDB as a standalone or clustered node, prompt for the address to which it will bind, and a password for the admin user. Responses to these prompts may be pre-seeded using standard debconf tools. Further details are in the README.Debian file.

For distributions lacking a compatible SpiderMonkey library, Apache CouchDB also provides packages for the 1.8.5 version.

Enabling the Apache CouchDB package repository

sudo apt update && sudo apt install -y curl apt-transport-https gnupg
curl https://couchdb.apache.org/repo/keys.asc | gpg --dearmor | sudo tee /usr/share/keyrings/couchdb-archive-keyring.gpg >/dev/null 2>&1
source /etc/os-release
echo "deb [signed-by=/usr/share/keyrings/couchdb-archive-keyring.gpg] https://apache.jfrog.io/artifactory/couchdb-deb/ ${VERSION_CODENAME} main" \
    | sudo tee /etc/apt/sources.list.d/couchdb.list >/dev/null

RedHat or CentOS: Run the following commands:

sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://couchdb.apache.org/repo/couchdb.repo

Installing the Apache CouchDB packages

sudo apt update
sudo apt install -y couchdb

Debian/Ubuntu installs from binaries can be pre-configured for single node or clustered installations. For clusters, multiple nodes will still need to be joined together and configured consistently across all machines; follow the Cluster Setup walkthrough to complete the process.

RedHat/CentOS: Run the command:

sudo yum install -y couchdb

Once installed, create an admin user by hand before starting CouchDB, if your installer didn’t do this for you already.

You can now start the service.

Your installation is not complete. Be sure to complete the Setup steps for a single node or clustered installation.

Relax! CouchDB is installed and running.

GPG keys used for signing the CouchDB repositories

pub   rsa8192 2015-01-19 [SC]
      390EF70BB1EA12B2773962950EE62FB37A00258D
uid           The Apache Software Foundation (Package repository signing key) <root@apache.org>

As of 2021.04.25, the package signing key (only used for rpm packages) is:

pub   rsa4096 2017-07-28 [SC] [expires: 2022-07-27]
      2EC788AE3F239FA13E82D215CDE711289384AE37
uid           Joan Touzet (Apache Code Signing Key) <wohali@apache.org>

Both are available from most popular GPG key servers.

Installation from source

This guide, as well as the INSTALL.Unix document in the official tarball release are the canonical sources of installation information. However, many systems have gotchas that you need to be aware of. In addition, dependencies frequently change as distributions update their archives.

Dependencies

It is recommended that you install Erlang OTP R16B03-1 or above where possible. You will only need libcurl if you plan to run the JavaScript test suite. And help2man is only need if you plan on installing the CouchDB man pages. Python and Sphinx are only required for building the online documentation. Documentation build can be disabled by adding the --disable-docs flag to the configure script.

Debian-based Systems

sudo apt-get --no-install-recommends -y install \
    build-essential pkg-config erlang \
    libicu-dev libmozjs185-dev libcurl4-openssl-dev

Be sure to update the version numbers to match your system’s available packages.

RedHat-based (Fedora, CentOS, RHEL) Systems

sudo yum install autoconf autoconf-archive automake \
    curl-devel erlang-asn1 erlang-erts erlang-eunit gcc-c++ \
    erlang-os_mon erlang-xmerl erlang-erl_interface help2man \
    libicu-devel libtool perl-Test-Harness

Warning: To build a release for CouchDB the erlang-reltool package is required, yet on CentOS/RHEL this package depends on erlang-wx which pulls in wxGTK and several X11 libraries. If CouchDB is being built on a console only server it might be a good idea to install this in a separate step to the rest of the dependencies, so that the package and all its dependencies can be removed using the yum history tool after the release is built. (reltool is needed only during release build but not for CouchDB functioning)

The package can be installed by running:

sudo yum install erlang-reltool

Mac OS X

If you are installing from source, you will need to install the Command Line Tools:

xcode-select --install

You can then install the other dependencies by running:

brew install autoconf autoconf-archive automake libtool \
    erlang icu4c spidermonkey curl pkg-config

You will need Homebrew installed to use the brew command.

Some versions of Mac OS X ship a problematic OpenSSL library. If you’re experiencing troubles with CouchDB crashing intermittently with a segmentation fault or a bus error, you will need to install your own version of OpenSSL. See the wiki, mentioned above, for more information.

SEE ALSO:

FreeBSD

You can install this by running:

pkg install gmake

Installing

./configure

If you wish to customize the installation, pass --help to this script.

If everything was successful you should see the following message:

You have configured Apache CouchDB, time to relax.

Relax.

To build CouchDB you should run:

make release

Try gmake if make is giving you any problems.

If include paths or other compiler options must be specified, they can be passed to rebar, which compiles CouchDB, with the ERL_CFLAGS environment variable. Likewise, options may be passed to the linker with the ERL_LDFLAGS environment variable:

make release ERL_CFLAGS="-I/usr/local/include/js -I/usr/local/lib/erlang/usr/include"

If everything was successful you should see the following message:

... done
You can now copy the rel/couchdb directory anywhere on your system.
Start CouchDB with ./bin/couchdb from within that directory.

Relax.

Note: a fully-fledged ./configure with the usual GNU Autotools options for package managers and a corresponding make install are in development, but not part of the 2.0.0 release.

User Registration and Security

You should create a special couchdb user for CouchDB.

On many Unix-like systems you can run:

adduser --system \
        --shell /bin/bash \
        --group --gecos \
        "CouchDB Administrator" couchdb

On Mac OS X you can use the Workgroup Manager to create users up to version 10.9, and dscl or sysadminctl after version 10.9. Search Apple’s support site to find the documentation appropriate for your system. As of recent versions of OS X, this functionality is also included in Server.app, available through the App Store only as part of OS X Server.

You must make sure that the user has a working POSIX shell and a writable home directory.

You can test this by:

As a recommendation, copy the rel/couchdb directory into /home/couchdb or /Users/couchdb.

Ex: copy the built couchdb release to the new user’s home directory:

cp -R /path/to/couchdb/rel/couchdb /home/couchdb

Change the ownership of the CouchDB directories by running:

chown -R couchdb:couchdb /home/couchdb

Change the permission of the CouchDB directories by running:

find /home/couchdb -type d -exec chmod 0770 {} \;

Update the permissions for your ini files:

chmod 0644 /home/couchdb/etc/*

First Run

You can start the CouchDB server by running:

sudo -i -u couchdb /home/couchdb/bin/couchdb

This uses the sudo command to run the couchdb command as the couchdb user.

When CouchDB starts it should eventually display following messages:

{database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users" ...

Don’t be afraid, we will fix this in a moment.

To check that everything has worked, point your web browser to:

http://127.0.0.1:5984/_utils/index.html

From here you should verify your installation by pointing your web browser to:

http://localhost:5984/_utils/index.html#verifyinstall

Your installation is not complete. Be sure to complete the Setup steps for a single node or clustered installation.

Running as a Daemon

The CouchDB team recommends runit to run CouchDB persistently and reliably. According to official site:

Configuration of runit is straightforward; if you have questions, contact the CouchDB user mailing list or IRC-channel #couchdb in FreeNode network.

Let’s consider configuring runit on Ubuntu 16.04. The following steps should be considered only as an example. Details will vary by operating system and distribution. Check your system’s package management tools for specifics.

Install runit:

sudo apt-get install runit

Create a directory where logs will be written:

sudo mkdir /var/log/couchdb
sudo chown couchdb:couchdb /var/log/couchdb

Create directories that will contain runit configuration for CouchDB:

sudo mkdir /etc/sv/couchdb
sudo mkdir /etc/sv/couchdb/log

Create /etc/sv/couchdb/log/run script:

#!/bin/sh
exec svlogd -tt /var/log/couchdb

Basically it determines where and how exactly logs will be written. See man svlogd for more details.

Create /etc/sv/couchdb/run:

#!/bin/sh
export HOME=/home/couchdb
exec 2>&1
exec chpst -u couchdb /home/couchdb/bin/couchdb

This script determines how exactly CouchDB will be launched. Feel free to add any additional arguments and environment variables here if necessary.

Make scripts executable:

sudo chmod u+x /etc/sv/couchdb/log/run
sudo chmod u+x /etc/sv/couchdb/run

Then run:

sudo ln -s /etc/sv/couchdb/ /etc/service/couchdb

In a few seconds runit will discover a new symlink and start CouchDB. You can control CouchDB service like this:

sudo sv status couchdb
sudo sv stop couchdb
sudo sv start couchdb

Naturally now CouchDB will start automatically shortly after system starts.

You can also configure systemd, launchd or SysV-init daemons to launch CouchDB and keep it running using standard configuration files. Consult your system documentation for more information.

Installation on Windows

Installation from binaries

WARNING:

NOTE:

NOTE:

Silent Install

Install CouchDB without a service, but with an admin user:password of admin:hunter2:

msiexec /i apache-couchdb-3.0.0.msi /quiet ADMINUSER=admin ADMINPASSWORD=hunter2 /norestart

The same as above, but also install and launch CouchDB as a service:

msiexec /i apache-couchdb-3.0.0.msi /quiet INSTALLSERVICE=1 ADMINUSER=admin ADMINPASSWORD=hunter2 /norestart

Unattended uninstall of CouchDB to target directory D:CouchDB:

msiexec /x apache-couchdb-3.0.0.msi INSTALLSERVICE=1 APPLICATIONFOLDER="D:\CouchDB" ADMINUSER=admin ADMINPASSWORD=hunter2 /quiet /norestart

Unattended uninstall if the installer file is unavailable:

msiexec /x {4CD776E0-FADF-4831-AF56-E80E39F34CFC} /quiet /norestart

Add /l* log.txt to any of the above to generate a useful logfile for debugging.

Installation from sources

Installation on macOS

Installation using the Apache CouchDB native application

That’s all, now CouchDB is installed on your Mac:

Installation with Homebrew

Installation from source

Running as a Daemon

The CouchDB team recommends runit to run CouchDB persistently and reliably. Configuration of runit is straightforward; if you have questions, reach out to the CouchDB user mailing list.

Naturally, you can configure launchd or other init daemons to launch CouchDB and keep it running using standard configuration files.

Consult your system documentation for more information.

Installation on FreeBSD

Installation from ports

cd /usr/ports/databases/couchdb
make install clean

This will install CouchDB from the ports collection.

NOTE:

Start script

couchdb_enable="NO"
couchdb_enablelogs="YES"
couchdb_user="couchdb"

After enabling the couchdb rc service use the following command to start CouchDB:

/usr/local/etc/rc.d/couchdb start

This script responds to the arguments start, stop, status, rcvar etc..

The start script will also use settings from the following config files:

Administrators should use default.ini as reference and only modify the local.ini file.

Post install

In case the install script fails to install a non-interactive user “couchdb” to be used for the database, the user needs to be created manually:

I used the pw command to add a user “couchdb” in group “couchdb”:

pw user add couchdb
pw user mod couchdb -c 'CouchDB, time to relax' -s /usr/sbin/nologin -d /var/lib/couchdb
pw group add couchdb

The user is added to /etc/passwd and should look similar to the following: