All jobs

Add Full-Text Search to a Mongoose API

Yes. An agent adds a MongoDB text index to a Mongoose model, a q parameter on the list route sorted by text score, Joi validation, tests and docs, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

Full-text search in MongoDB is a text index plus a $text filter, so the agent first reads the model and the query path that will carry it. In an API laid out like hagopj13/node-express-boilerplate, package.json depends on mongoose ^5.7.7, resolved to 5.13.20 in yarn.lock, with express 4.17.1, joi ^17.3.0, jest 26.6.3 and supertest 6.1.3, and its test script is jest -i --colors --verbose --detectOpenHandles. src/models/user.model.js declares name (trimmed, required), email (unique, lowercased, checked with validator.isEmail), password (marked private for the toJSON plugin), role and isEmailVerified, with timestamps: true, and applies two plugins from src/models/plugins/: toJSON, which strips __v, timestamps and private paths and renames _id to id, and paginate, which adds a static paginate(filter, options) that builds a sort string from sortBy, defaults limit to 10 and page to 1, runs countDocuments and find().sort().skip().limit() in parallel, and returns results, page, limit, totalPages and totalResults. There is no index declaration on the schema beyond unique on email, and no $text anywhere.

The list route is GET /v1/users in src/routes/v1/user.route.js, guarded by auth('getUsers') and validate(userValidation.getUsers), whose Joi schema in src/validations/user.validation.js allows name, role, sortBy, limit and page. src/controllers/user.controller.js picks name and role into a filter and sortBy, limit and page into options with src/utils/pick.js, and src/services/user.service.js hands both to User.paginate. So name today is an exact match, and a search for part of a name returns nothing. src/config/config.js builds the Mongo URL from MONGODB_URL and appends -test under NODE_ENV=test, and tests/utils/setupTestDB.js connects with the useCreateIndex option and empties every collection before each test, which leaves indexes in place.

The agent adds one line to the schema, userSchema.index({ name: 'text', email: 'text' }), with a weight that favours name, since a collection takes one text index and both fields belong in it. It adds q: Joi.string().trim().min(1) to the getUsers validation, picks q in the controller and turns it into filter.$text = { $search: q }, and teaches paginate to handle a text filter: when filter.$text is set and no sortBy is given, the query projects score: { $meta: 'textScore' } and sorts by that score, so the best match comes first instead of createdAt. The toJSON plugin would otherwise expose score, so the agent drops it in the transform. The Swagger block above the route in user.route.js gains a q query parameter, since src/docs/swaggerDef.js builds the /v1/docs page from those comments. queryUsers in the service does not change; User.paginate already receives whatever the controller built.

To run the suite the agent needs a MongoDB. The sandbox has no Docker, so it installs it (sudo apt-get install -y docker.io, then starts dockerd) and runs the mongo:4.2.1-bionic image that docker-compose.yml names on port 27017, then exports the variables .env.example lists with MONGODB_URL pointing at it. It adds tests to tests/integration/user.test.js using insertUsers from tests/fixtures/user.fixture.js: after await User.init() so the text index has been built, a request with q set to one user's first name returns that user first and not the others, a q matching nothing returns totalResults: 0, and an empty q is a 400 from Joi. It adds a unit test to tests/unit/models/plugins/paginate.plugin.test.js for the score sort, and runs yarn lint, which applies the Airbnb and Prettier rules in .eslintrc.json, before yarn test.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add full-text search to the users API in this Express and Mongoose app. Read package.json, src/models/user.model.js, src/models/plugins/paginate.plugin.js, src/models/plugins/toJSON.plugin.js, src/services/user.service.js, src/controllers/user.controller.js, src/validations/user.validation.js, src/routes/v1/user.route.js, src/config/config.js, tests/utils/setupTestDB.js, tests/fixtures/user.fixture.js and docker-compose.yml first. Add a text index on name and email to the user schema with name weighted higher, accept a q query parameter on GET /v1/users validated with Joi, turn it into a $text filter in the controller, and make the paginate plugin sort by textScore when a text filter is present and no sortBy is given, without exposing the score in responses. Update the Swagger comment for the route. Install Docker in the sandbox, start the mongo image docker-compose.yml names, set the variables from .env.example, add integration tests for a matching search, an empty result and an empty q, add a unit test for the score sort, and run yarn lint and yarn test. Open a pull request that explains how the tokeniser treats an email address and shows the test output.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

The pull request touches src/models/user.model.js, src/models/plugins/paginate.plugin.js, src/models/plugins/toJSON.plugin.js, src/validations/user.validation.js, src/controllers/user.controller.js and src/routes/v1/user.route.js, and adds tests in tests/integration/user.test.js and tests/unit/models/plugins/paginate.plugin.test.js. Its description shows a GET /v1/users?q= request and the response order, explains that the text index is built by Mongoose on connect because autoIndex is on, and pastes the yarn test summary from the sandbox.

The repository has a .travis.yml and no GitHub Actions workflow, so GitHub shows no check on the pull request unless the team has added one; the run's record is where a reviewer reads the yarn lint and yarn test output, including the Mongo container start. A reviewer checks that the index weight matches what the product wants, that score never reaches a response, and that the empty-q behaviour is a 400 rather than an unfiltered list.

Before merging on a database with existing users, a reviewer notes that Mongoose creates the index when the app connects, which on a large collection takes time and holds a build lock on older MongoDB versions; the pull request says to create it with createIndex during a quiet window first, after which the connect-time call finds it already there.

Run it on every Linear issue labelled search

A team that files search requests as Linear issues runs this prompt from an automation in the Project with a linear trigger on issues labelled search. The trigger adds the issue after the prompt, so a request to search posts instead of users reaches the agent with the model and fields named in the issue, and the run reads that model's schema and routes before writing anything. One run starts per labelled issue, and the pull request links back to it.

An organisation with several Mongoose services starts one run per repository from its own server with POST /o/{org}/sessions, the repository in resources and a request_id that is a UUID v5 of the repository's full name, so a retry returns the run already started. Each run reads its own schema, so a service that already has a text index gets a search parameter wired to it, and one that searches on a field with an array of strings gets the index declared on that path.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

A text index is one line, but the behaviour around it is where mistakes hide: a second text index is refused by MongoDB, a $text filter with the plugin's default createdAt sort returns the newest match rather than the best, and the score projection leaks through toJSON unless it is stripped. The agent runs the integration tests against a real MongoDB in its own sandbox, so the pull request shows the index building, the sort order and the response shape rather than describing them.

The run keeps every command, from the Docker install to yarn test, so a reviewer who cannot start a Mongo locally still sees the suite pass. The same prompt runs across every service that uses this boilerplate at once, each with its own database and its own pull request.

Frequently asked questions

Why a MongoDB text index rather than Atlas Search?

Because this app points at any MongoDB through MONGODB_URL, including the mongo:4.2.1-bionic image in its compose file, and a text index works on all of them with no extra service. Atlas Search needs an Atlas cluster and an aggregation stage; a team on Atlas can swap the $text filter for $search later without touching the route or the validation.

Does searching an email address work?

The tokeniser splits on whitespace and punctuation, so a search for the part before the @ matches, and a full address matches its parts. The pull request shows what a search for a full address and for a domain returns against the test fixtures, so the behaviour is known before anyone relies on it.

Can I still filter by role and sort by name while searching?

Yes. role stays an exact filter and combines with $text, and an explicit sortBy such as name:asc takes precedence over the score sort. Only when q is given and sortBy is not does the plugin sort by textScore.

What happens to the index when the collection is large?

Mongoose calls createIndex on connect because autoIndex is on by default, and the first build on a large collection takes time. The pull request recommends building it ahead of the deploy with createIndex and keeping autoIndex on so the app finds it in place.

More Mongoose jobs

The same job on other stacks

Keep reading