Contact Us

Home / Engineering Blog / Security

SecurityEngineering notes

How GitHub Uses CodeQL for Security

This article covers how GitHub's Product Security Engineering team manages the rollout of CodeQL across the whole company — and how you can do the same.

GitHub's Product Security Engineering team writes code and ships tools that help keep the code underlying GitHub secure. We use GitHub Advanced Security (GHAS) to find, track, and fix vulnerabilities, and to enforce secure coding standards across the whole company. One of our key tools for large-scale code analysis is CodeQL.

CodeQL is GitHub's static analysis engine that runs automated security analysis. You can run queries against code the same way you run queries against a database. That is a much more powerful way to analyze code and find issues than traditional text search across a codebase.

This article walks through how we use CodeQL to keep GitHub secure and how you can apply the same lessons to your project. You will see why and how we use:

  • Custom query packs (and how we create and manage them). Custom queries.
  • Variant analysis for finding potentially unsafe coding practices.

Scaling CodeQL

We at GitHub use CodeQL in several ways.

  1. Default setup with the default and security-extended query suites. Default setup with the default and security-extended query suites covers the needs of the overwhelming majority of our more than 10,000 repositories. With that configuration, pull requests are automatically checked for security by CodeQL.
  2. Advanced setup with a custom query pack. Some repositories — such as our large Ruby monolith — need extra attention, so we use advanced setup with a query pack attached to tailor the analysis to our needs as closely as possible.
  3. Variant analysis across multiple repositories (MRVA). We use MRVA for variant analysis and quick audit checks. We also write our own CodeQL queries to find patterns in the code that are either specific to GitHub's codebase or require manual review by a security engineer.

The specific step of our custom Actions workflow that we use for the monolith is straightforward. It looks like this:

- name: Initialize CodeQL
    uses: github/codeql-action/init@v3
    with:
      languages: ${{ matrix.language }}
      config-file: ./.github/codeql/${{ matrix.language }}/codeql-config.yml

Our Ruby configuration is standard, but advanced setup allows a variety of options through custom configuration files. The interesting part here is the packs parameter, which pulls our custom query pack into the CodeQL analysis. That pack contains a set of CodeQL queries we wrote for Ruby, tailored to GitHub's codebase.

Here is why we did it — and how.

Publishing the CodeQL query pack

At first we kept CodeQL query files directly in GitHub's monolith repository, but we later dropped that approach for several reasons:

  • Every new or updated query had to go through the production deployment process.
  • Queries not included in a pack were not compiled ahead of time, which slowed down CodeQL analysis runs in CI.
  • Our CodeQL query test suite ran as part of the monolith's CI jobs. When a new CodeQL CLI version came out, query tests would sometimes fail because of changes in query execution results, even if the code in the pull request had not changed. This often caused confusion and frustration for engineers, since the failure had nothing to do with the changes they were making.

By moving to publishing the query pack in the GitHub Container Registry (GCR), we simplified the workflow and eliminated many of these problems, which made developing and maintaining CodeQL queries easier. So while it is still possible to keep custom queries directly in a repository, we recommend publishing queries as a pack in GCR to simplify deployment and speed up iteration.

Building the query pack

When building a custom query pack we had to account for a few important considerations, especially around dependency management — the ruby-all pack, for example.

To keep custom queries maintainable and concise, we extend classes from the standard query library, such as ruby-all. That lets us reuse existing functionality instead of reinventing the wheel, keeping queries compact and easy to maintain. At the same time, changes to CodeQL library APIs can introduce breaking changes, which can make queries outdated or cause errors. Since CodeQL runs inside the CI system, we work to minimize the chance of such failures, because they frustrate developers and erode trust.

We develop queries against the latest version of the ruby-all pack so we always have current functionality. To reduce the risk that a dependency change will hurt CI, we pin the ruby-all version in codeql-pack.lock.yml before release. That guarantees the queries run against the tested ruby-all version once deployed, avoiding problems caused by unintended upgrades.

Here is how we manage this setup:

  • In qlpack.yml we declare a dependency on the latest ruby-all version.
  • During development this configuration pulls the latest ruby-all version when running codeql pack init, which keeps us on the current version.
// qlpack.yml file for our custom query pack

library: false
name: github/internal-ruby-codeql
version: 0.2.3
extractor: 'ruby'
dependencies:
  codeql/ruby-all: "*"
tests: 'test'
description: "Ruby CodeQL queries used internally at GitHub"

Before release we pin the version in the codeql-pack.lock.yml file, specifying an exact version to keep things stable and prevent CI problems.

// codeql-pack.lock.yml file for our custom query pack

lockVersion: 1.0.0
dependencies:
 ...
 codeql/ruby-all:
   version: 1.0.6

This approach balances developing against the newest ruby-all features with stability at release time.

We also have a set of CodeQL unit tests that check queries against code examples. That makes it possible to tell quickly whether a query will error out even before the pack is published. These tests run as part of CI in the query pack repository, which catches errors early. We strongly recommend writing unit tests for custom CodeQL queries to keep them stable and reliable.

In general, the basic process for releasing new CodeQL queries through a pack looks like this:

  • Open a pull request with a new query.
  • Write unit tests for the new query.
  • Merge the pull request.
  • Bump the pack version in a new pull request.
  • Run codeql pack init to resolve dependencies.
  • Adjust the unit tests if needed. Publish the query pack to GitHub Container Registry (GCR).
  • Repositories that use this pack in their configuration start picking up the updated queries.

We found that this process strikes a good balance between development convenience for our team and stability of the released query pack.

Configuring a repository to use a custom query pack

We are not going to give universal configuration recommendations, because in the end it depends on how code deployment works in your organization. We chose not to pin a specific pack version in the CodeQL configuration file (see above). Instead, we manage versioning by publishing the CodeQL pack to GCR. This lets GitHub's monolith automatically pick up the latest published version of the query pack. To roll back a change, it is enough to publish a new pack version. For example, in one case we released a query with a lot of false positives and were able to publish a new pack version with that query removed in under 15 minutes. That is faster than it would have taken to open and merge a pull request bumping the version in the CodeQL configuration file.

One problem we ran into when publishing the query pack to GCR was finding a convenient way to give many repositories across the company access to the pack. We considered several options:

  • Grant access to individual repositories. On the package management page you can assign permissions for each repository manually. With our number of repositories that turned out to be impractical, and there is currently no way to automate the process through the API.
  • Issue a personal access token for the CodeQL Action runner. We could create a personal access token (PAT) with read access to all packages in our organization and add it to the runner. That would mean managing a new token, and the access level seemed too broad to us, since it would allow reading all of our private packages, not only the ones that need access.
  • Grant access through a linked repository. We ended up implementing the third option. We link a repository to the package and let the package inherit access permissions from the linked repository.

Custom queries in the CodeQL pack

We write a variety of custom queries for use in our custom packs. They cover GitHub-specific patterns that are not part of the standard CodeQL query pack. This lets us tailor the analysis to the specifics of our company and codebase. Examples of the kinds of checks we run with our custom query pack:

  • Use of high-risk APIs specific to GitHub's code that can be dangerous when handling untrusted user input.
  • Use of certain built-in Rails methods for which we have safer in-house alternatives.
  • Missing mandatory authorization methods in REST API endpoint definitions and GraphQL object/mutation definitions.
  • REST API endpoints and GraphQL mutations where access control methods must be explicitly defined to check user permissions. (Specifically: the query finds the absence of such methods to make sure endpoint access rights are checked.)
  • Use of signed tokens, to remind developers to bring in Product Security as reviewers when they use them.

Custom queries can serve purposes beyond blocking code releases — they are also educational. For example, we want to notify engineers when the ActiveRecord::decrypt method is used. That method generally should not be used in production code, because it decrypts encrypted columns and saves them in unencrypted form. In the metadata of such a query we use the "recommendation" severity level, so these warnings are informational. They can show up in the pull request, but they do not fail the CodeQL CI job.

Using a lower severity level lets engineers judge the impact of new queries themselves without immediate intervention. Also, such warnings are not tracked within our Fundamentals program, which means they do not require urgent fixes — reflecting the maturity of the query as its relevance and risk assessment are refined further.

/**
 * @id rb/github/use-of-activerecord-decrypt
 * @description Do not use the `.decrypt` method on ActiveRecord models. It will decrypt all encrypted attributes
 * and save them in unencrypted form, which effectively undoes the encryption and can make data inaccessible.
 * If you need the unencrypted value of a specific attribute, use the call my_model.attribute_name.
 * @kind problem
 * @severity recommendation
 * @name Use of the decrypt method in ActiveRecord
 * @tags security
 *       github-internal-use
 */

import ruby
import DataFlow
import codeql.ruby.DataFlow
import codeql.ruby.frameworks.ActiveRecord

/** Finds `.decrypt` method calls where the receiver may be an ActiveRecord object */
class ActiveRecordDecryptMethodCall extends ActiveRecordInstanceMethodCall {
  ActiveRecordDecryptMethodCall() { this.getMethodName() = "decrypt" }
}

from ActiveRecordDecryptMethodCall call
select call,
  "Не используйте метод `.decrypt` у моделей ActiveRecord — это расшифрует все зашифрованные атрибуты и сохранит их в незашифрованном виде."

Another educational query is the one mentioned above, which finds a missing control_access method in a class defining a REST API endpoint. If a pull request adds a new endpoint without control_access, a comment is automatically posted saying that the control_access method was not found and that it is required for REST API endpoints. This notifies the reviewer of a potential problem and reminds the developer to fix it.

/**
 * @id rb/github/api-control-access
 * @name REST API without 'control_access'
 * @description All REST API endpoints must call the 'control_access' method to guarantee access only for the specified actor types.
 * @kind problem
 * @tags security
 *       github-internal-use
 * @precision high
 * @problem.severity recommendation
 */

import codeql.ruby.AST
import codeql.ruby.DataFlow
import codeql.ruby.TaintTracking
import codeql.ruby.ApiGraphs

// REST API endpoints in the Api::App module generally must call the control_access method
private DataFlow::ModuleNode appModule() {
  result = API::getTopLevelMember("Api").getMember("App").getADescendentModule() and
  not result = protectedApiModule() and
  not result = staffAppApiModule()
}

// REST API endpoints in the Api::Admin, Api::Staff, Api::Internal and Api::ThirdParty modules do not have to call the control_access method
private DataFlow::ModuleNode protectedApiModule() {
  result =
    API::getTopLevelMember(["Api"])
        .getMember(["Admin", "Staff", "Internal", "ThirdParty"])
        .getADescendentModule()
}

// REST API endpoints in the Api::Staff::App module also do not have to call the control_access method
private DataFlow::ModuleNode staffAppApiModule() {
  result =
    API::getTopLevelMember(["Api"]).getMember("Staff").getMember("App").getADescendentModule()
}

// Class describing a REST API route where the control_access method was not called
private class ApiRouteWithoutControlAccess extends DataFlow::CallNode {
  ApiRouteWithoutControlAccess() {
    this = appModule().getAModuleLevelCall(["get", "post", "delete", "patch", "put"]) and
    not performsAccessControl(this.getBlock())
  }
}

// Predicate determining whether an access control method was called in the block
predicate performsAccessControl(DataFlow::BlockNode blocknode) {
  accessControlCalled(blocknode.asExpr().getExpr())
}

// The control_access method is called somewhere inside the given block
predicate accessControlCalled(Block block) {
  block.getAStmt().getAChild*().(MethodCall).getMethodName() = "control_access"
}

// The query returns the REST API endpoints where the control_access method was not called
from ApiRouteWithoutControlAccess api
select api.getLocation(),
  "Метод control_access не был обнаружен в этом REST API эндпоинте. Все такие эндпоинты должны вызывать этот метод, чтобы гарантировать доступ только для определённых типов акторов."

Variant analysis

Variant analysis (Variant Analysis, VA) is the process of finding variants of known vulnerabilities. This method is especially useful when we respond to a bug bounty report or a security incident. For that we use a combination of tools, including GitHub code search, our own scripts, and CodeQL. We usually start with code search to find patterns similar to the one that caused a specific vulnerability across various repositories. Sometimes that is not enough, because code search is semantically insensitive — it cannot tell whether a variable is an Active Record object or whether it is used in an if expression. To answer questions like those, we use CodeQL.

When we write CodeQL queries for variant analysis, we care much less about the number of false positives, because the main goal is to hand results over to security engineers for further analysis. Code quality also matters less, since the queries are only used within a specific VA exercise. Here are examples of tasks we use CodeQL for in the VA process:

  • Where is SHA1 hashing used in the code?
  • A recent bug bounty report said one of our internal API endpoints was vulnerable to SQL injection. Where do we pass user input into that endpoint?
  • Some Ruby HTTP request libraries have a problem with proxy configuration handling. Can we find places where we instantiate such libraries with a proxy set?

Recently we investigated a subtle vulnerability in Rails. We needed to find the following condition in the code:

  • A parameter is used to look up an Active Record object.
  • That parameter is then reused after the Active Record lookup.

The danger is that this can lead to an IDOR vulnerability (Insecure Direct Object Reference), because Active Record finder methods can accept an array. If the code first looks up an Active Record object to check access to a resource, then reuses a different element from that array to look up an object, it can lead to an IDOR. Writing a query that reliably catches all vulnerable cases of this pattern would be hard, but we managed to create a query that finds potential vulnerabilities and outputs a list of code paths for manual review. We ran this query across a large number of our Ruby codebases using MRVA in CodeQL.

The query itself — fairly rough and not production quality — is below:

/**
 * @name draft query: array passed to AR
 * @description an array is passed to an ActiveRecord finder object
 */

import ruby
import codeql.ruby.AST
import codeql.ruby.ApiGraphs
import codeql.ruby.frameworks.Rails
import codeql.ruby.frameworks.ActiveRecord
import codeql.ruby.frameworks.ActionController
import codeql.ruby.DataFlow
import codeql.ruby.Frameworks
import codeql.ruby.TaintTracking

// Gets the "ultimate" receiver in a chain of method calls.
// For example, in `Foo.bar` that is the access to `Foo`, and in
// `foo.bar.baz("arg")` it is the variable `foo`
private Expr getUltimateReceiver(MethodCall call) {
  exists(Expr recv |
    recv = call.getReceiver() and
    (
      result = getUltimateReceiver(recv)
      or
      not recv instanceof MethodCall and result = recv
    )
  )
}

// Method names on ActiveRecord model classes that may return
// one or more model instances. Also includes the `initialize` method.
// See https://api.rubyonrails.org/classes/ActiveRecord/FinderMethods.html
private string staticFinderMethodName() {
  exists(string baseName |
    baseName = ["find_by", "find_or_create_by", "find_or_initialize_by", "where"] and
    result = baseName + ["", "!"]
  )
  // or:
  // result = ["new", "create"]
}

// Call to an ActiveRecord model instance finder method
private class ActiveRecordModelFinderCall extends ActiveRecordModelInstantiation, DataFlow::CallNode
{
  private ActiveRecordModelClass cls;

  ActiveRecordModelFinderCall() {
    exists(MethodCall call, Expr recv |
      call = this.asExpr().getExpr() and
      recv = getUltimateReceiver(call) and
      (
        // The receiver is a reference to an ActiveRecord model class by name
        recv.(ConstantReadAccess).getAQualifiedName() = cls.getAQualifiedName()
        or
        // The receiver is self, and the call happens inside a singleton method
        // of the ActiveRecord model
        recv instanceof SelfVariableAccess and
        exists(SingletonMethod callScope |
          callScope = call.getCfgScope() and
          callScope = cls.getAMethod()
        )
      ) and
      (
        call.getMethodName() = staticFinderMethodName()
        or
        // dynamically generated finder methods
        call.getMethodName().indexOf("find_by_") = 0
      )
    )
  }

  final override ActiveRecordModelClass getClass() { result = cls }
}

// Argument passed to the model finder method
class FinderCallArgument extends DataFlow::Node {
  private ActiveRecordModelFinderCall finderCallNode;

  FinderCallArgument() { this = finderCallNode.getArgument(_) }
}

// Reference to a value from the params hash
class ParamsHashReference extends DataFlow::CallNode {
  private Rails::ParamsCall params;

  // TODO: for now only direct accesses to params are considered
  ParamsHashReference() { this.getReceiver().asExpr().getExpr() = params }

  string getArgString() {
    result = this.getArgument(0).asExpr().getConstantValue().getStringlikeValue()
  }
}

// Configuration tracking data flow from the parameter array into finder methods
class ArrayPassedToActiveRecordFinder extends TaintTracking::Configuration {
  ArrayPassedToActiveRecordFinder() { this = "ArrayPassedToActiveRecordFinder" }

  override predicate isSource(DataFlow::Node source) { source instanceof ParamsHashReference }

  override predicate isSink(DataFlow::Node sink) {
    sink instanceof FinderCallArgument
  }

  string getParamsArg(DataFlow::CallNode paramsCall) {
    result = paramsCall.getArgument(0).asExpr().getConstantValue().getStringlikeValue()
  }

  // Does not handle complex cases (e.g. if/else),
  // Meant for quick manual filtering of interesting cases,
  // so it works as generally as possible to avoid missing cases
  predicate paramsUsedAfterLookups(DataFlow::Node source) {
    exists(DataFlow::CallNode y | y instanceof ParamsHashReference
    and source.getEnclosingMethod() = y.getEnclosingMethod()
    and source != y
    and getParamsArg(source) = getParamsArg(y)
    // we only care about reuses after the object lookup
    and y.getLocation().getStartLine() > source.getLocation().getStartLine())
  }
}

// Selects sources and sinks where there is data flow from params into a model finder call,
// and the same parameters are reused after the lookup
from ArrayPassedToActiveRecordFinder config, DataFlow::Node source, DataFlow::Node sink
where config.hasFlow(source, sink) and config.paramsUsedAfterLookups(source)
select source, sink.getLocation()

Conclusion

CodeQL can be extremely useful for product security teams to find and prevent vulnerabilities across an entire codebase. We use a combination of queries that run in CI through a query pack, plus one-off queries run with MRVA, to find potential vulnerabilities and pass information about them to engineers. But CodeQL is not only useful for finding security vulnerabilities — it also helps detect the presence or absence of security measures defined in the code. That saves our security team time by automatically finding some issues, and saves engineers time by letting them catch issues earlier in development.

← Back to all engineering notes