{"componentChunkName":"component---src-components-mark-down-block-js","path":"/docs/crash-analysis/","result":{"data":{"markdownRemark":{"html":"

Analyzing Application Crashes

\n

When an application crashes, you can use AppScope together with eBPF to generate useful files and send those files to a network destination, before the environment is torn down.

\n

In the walkthrough that follows, we'll run Nginx in a Docker container, force Nginx to crash, and send crash data – backtrace, core dump, and snapshot files – through Cribl.Cloud to Amazon S3.

\n

If you prefer, you can easily adapt the procedure to analyze a crash of any process (not just Nginx); run on the host rather than a container; or, route the crash data to any network destination (not just Cribl.Cloud). You can even skip the network destination entirely and just view the crash files on disk.

\n

Prerequisites

\n

You will need:

\n\n

Setting Up Your Cribl.Cloud Instance

\n

In Cribl.Cloud, start by creating an Amazon S3 Destination:

\n
    \n
  1. Click Manage Stream.
    2. \n
  2. Under Worker Groups, click default.
    3. \n
  3. Click Data > Destinations and choose the Amazon S3 Destination.
    4. \n
  4. Click Add Destination and configure the Destination with your S3 credentials and preferences.
    5. \n
  5. Click Save.
    6. \n
\n

Next, configure the built-in TCP Source:

\n
    \n
  1. Click Data > Sources and choose the TCP Source.
    2. \n
  2. Click the built-in in_tcp Source, enable it, and configure it with address 0.0.0.0 and port 10060.
    3. \n
  3. Click Connected Destinations and choose QuickConnect. Select your S3 Destination.
    4. \n
  4. Click Save.
    5. \n
  5. Click Commit and Deploy.
    6. \n
\n

You now have a Cribl.Cloud Worker listening for TCP data, and configured to send it on to S3.

\n

Setting Up AppScope and eBPF

\n

The scope-ebpf and AppScope projects are separate and independent, and they operate under different licenses – for that reason, each has its own repo.

\n

Download the AppScope binary:

\n
cd ~/Downloads  \ncurl -Lo scope https://cdn.cribl.io/dl/scope/1.4.1/linux/$(uname -m)/scope\ncurl -Ls https://cdn.cribl.io/dl/scope/1.4.1/linux/$(uname -m)/scope.md5 | md5sum -c \nchmod +x scope\n
\n

Download and build the scope-ebpf binary:

\n
git clone git@github.com:criblio/scope-ebpf.git\ncd scope-ebpf\nmake all\n
\n

Deploy the eBPF module to listen for crashing applications:

\n
sudo ~/Downloads/scope-ebpf/bin/scope-ebpf\n
\n

Leave this shell running and open another terminal or tab.

\n

Deploy the AppScope daemon, to receive messages from eBPF and send crash files to a network destination:

\n
sudo ~/Downloads/scope daemon --filedest tcp://<path-to-cribl-cloud-tcp>:10060\n
\n

Once you start the daemon, the scope-ebpf binary will exit in the other terminal. That's normal.

\n

At this point, preparation is complete:

\n\n

Making an Application Crash

\n

Start a daemonized Nginx container:

\n
docker run --rm -d nginx\n
\n

Attach AppScope to the containerized Nginx process, and enable backtrace and core dump on crash:

\n
sudo ~/Downloads/scope attach --backtrace --coredump nginx\n
\n

This command shows the Nginx process with its child processes, in a list. Note the PID of the parent Nginx process.

\n

(Alternatively, you could have used the scope run command or the scope rules command to tell AppScope to load itself into an application when it starts.)

\n

Force a crash of Nginx by sending the process a BUS error signal:

\n
sudo kill -s SIGBUS <PID_of_nginx>\n
\n

AppScope will respond to any of the usual signals of a crashing process: SIGBUS, SIGINT, SIGSEGV, or SIGFPE.

\n

Now look in your S3 bucket – you should see four files: info, backtrace, snapshot, and cfg. The same files, plus the core dump, should also be in /tmp/appscope/<PID_of_nginx>/.

\n

Investigating the Crash

\n

Here are a few hints about the meaningful insights that the crash files provide.

\n

The backtrace file contains the application stack trace at the time of the crash:

\n

\n \n \n \n \n

\n

The snapshot file contains properties of the process:

\n

\n \n \n \n \n

\n

The core dump (not shown here) contains a binary snapshot of the application at the time of the crash, for inspection with a debugger like gdb. By default, the daemon does not send this over the network since it's not usually useful outside of its origin environment. The man page on core dumps can help you explore this data.

\n

If this topic is of interest to you, and/or if you'd like some help getting the procedure to work, please get in touch – there are a variety of ways to do that.

","frontmatter":{"title":"Analyzing Application Crashes"}}},"pageContext":{"slug":"/crash-analysis/","id":"5857d3f7-b3d8-50c6-b2c2-cb6fc7701ed0"}},"staticQueryHashes":["2787008656","306729481","625673991","640077088"]}