Segmentation fault when running pip inside docker buildx on host=arm platform=amd64

Following Docker's transition to a paid licensing model for Docker Desktop, I migrated our development environment to Rancher Desktop, which offers integrated support for both Docker and Kubernetes functionalities. During a recent initiative to convert legacy Docker images to multi-platform images, I encountered an unusual issue.

During the buildx build, the `pip install` command was failing with a segmentation fault error, despite the same installation steps executing flawlessly in an interactive container environment.

Initial Troubleshooting Efforts

Further investigation revealed:

1. The ARM64 architecture builds completed successfully, while AMD64 builds consistently failed

2. The failures were isolated to two specific Python packages: pycrypto and uWSGI

3. Standard remediation approaches suggested on various posts were ineffective, including:

   - Modifying Python versions

   - Updating pip

   - Testing alternative package versions

   - Installing supplementary system libraries

   - Switching between base images (python:3.8-slim (Debian bookworm), Debian bullseye, Ubuntu)

   - Using root installation instead of virtual environments

Root Cause Analysis

After extensive troubleshooting, I identified that the failures stemmed from cross-architecture compilation requirements. Both uWSGI and pycrypto lack pre-built wheels, necessitating compilation during installation. This compilation process was failing specifically when building AMD64 binaries on ARM64 host architecture.

Shifting my focus to the underlying virtualization platform, I searched for issues related to QEMU emulation and segmentation faults on AMD64 platforms. A comment on a relevant post suggested trying the build process on Docker Desktop. 

Resolution

After reinstalling Docker Desktop, I was able to successfully build the image.

To verify the findings, I retested all previous versions of the Dockerfile, and they worked as expected.

Conclusion

The root cause of this issue lies in the QEMU virtualization framework's emulation of AMD64 architectures on ARM-based systems. 

While ARM support has matured considerably across the board, such cases represent areas requiring further development attention from the community. 

NOTE

To facilitate easy reuse and streamline the build process, I recommend creating a builder

docker buildx create \

--name multi-platform-docker \

--driver docker-container \

--platform linux/amd64,linux/arm64

To utilize this builder, run the following command:

docker buildx --builder multi-platform-docker build .

Multi-platform multi-stage build docker images with platform specific files

 While working on multi platform docker images for a FastAPI application with MySQL client libraries, ran into a scenario where the MySQL library files are stored in platform specific paths.

/usr/lib/x86_64-linux-gnu

/usr/lib/aarch64-linux-gnu

As part of multi-stage build, I wanted to do something like

FROM python:${PYTHON_VERSION}-slim

# if x64

COPY --from=builder /usr/lib/x86_64-linux-gnu/libmysql* /usr/lib/x86_64-linux-gnu

# if arm

COPY --from=builder /usr/lib/aarch64-linux-gnu/libmysql* /usr/lib/aarch64-linux-gnu

But there is no conditional platform support at the COPY command level. It is only at the FROM level.

The simple solution would be to have duplicate final image setup.

FROM --platform=amd64 python:${PYTHON_VERSION}-slim

COPY --from=builder /usr/lib/x86_64-linux-gnu/libmysql* /usr/lib/x86_64-linux-gnu
...

FROM --platform=arm64 python:${PYTHON_VERSION}-slim

COPY --from=builder /usr/lib/aarch64-linux-gnu/libmysql* /usr/lib/aarch64-linux-gnu
...

But this would mean duplicating rest of the steps too.

The solution is to create multiple platform specific intermediate images and use them to create the final image.

FROM --platform=amd64 python:${PYTHON_VERSION}-slim as final-intermediate-amd64

COPY --from=builder /usr/lib/x86_64-linux-gnu/libmysql* /usr/lib/x86_64-linux-gnu
COPY --from=builder /usr/lib/x86_64-linux-gnu/libmaria* /usr/lib/x86_64-linux-gnu
COPY --from=builder /usr/lib/x86_64-linux-gnu/*xslt* /usr/lib/x86_64-linux-gnu
COPY --from=builder /usr/lib/x86_64-linux-gnu/*xml* /usr/lib/x86_64-linux-gnu

FROM --platform=arm64 python:${PYTHON_VERSION}-slim as final-intermediate-arm64

COPY --from=builder /usr/lib/aarch64-linux-gnu/libmysql* /usr/lib/aarch64-linux-gnu
COPY --from=builder /usr/lib/aarch64-linux-gnu/libmaria* /usr/lib/aarch64-linux-gnu
COPY --from=builder /usr/lib/aarch64-linux-gnu/*xslt* /usr/lib/aarch64-linux-gnu
COPY --from=builder /usr/lib/aarch64-linux-gnu/*xml* /usr/lib/aarch64-linux-gnu

FROM final-intermediate-${TARGETARCH}

COPY . /code/
COPY requirements.txt /code/requirements.txt
COPY --from=builder /code/.venv /code/.venv
COPY --from=builder /usr/bin/mysql_config /usr/bin

WORKDIR /code

EXPOSE 8000

CMD ["/code/.venv/bin/uvicorn", "--app-dir", "/code/myapp", "server:app", "--proxy-headers"]

FastAPI middleware performance

 As per the FastAPI docs, the way to create and add custom middlewares is

@app.middleware("http")

async def add_my_middlware(request: Request, call_next):

response = await call_next(request)

return response

Seems simple enough. Before the await, you can do something with the request. After the await, you can do something with the response.

But if you run benchmark, you will find something very surprising.

Python - calling async function from a sync code flow

 Recently I ran into a scenario where I needed to call a init function for a third party library in my FastAPI application. The problem was, the function was async.

One way of calling an async function from a sync flow is using asyncio event loop.

import asyncio

loop = asyncio.get_event_loop()

loop.run_until_complete(init())

But this gave event loop is already running error.

The problem is, FastAPI application is run with uvicorn which starts a loop.

So I tried creating a new loop.

loop = asyncio.new_event_loop()

asyncio.set_event_loop(loop)

loop.run_until_complete(init())

But this still didn't work as asyncio only supports one loop at a time.

The most suggested approach is to use nest-asyncio. 

import nest_asyncio

loop = asyncio.new_event_loop()

nest_asyncio.apply(loop)

asyncio.set_event_loop(loop)

loop.run_until_complete(init())

This raised an exception: Can't patch uvloop.Loop.

uvicorn patches asyncio to use uvloop which is better performant that vanilla asyncio (previous claims were 2x to 4x. In the simple test I ran, even with performance changes in 3.12, asyncio was about 25% slower than uvloop.).

Did some research, but couldn't find any solution around this other than forcing uvicorn to run with vanilla asyncio

uvicorn main:app --loop asyncio

It didn't make sense to take a performance hit, just to call a function.

So I decided to dig deeper into why asyncio.new_event_loop returns uvloop.Loop.

The way uvloop does it is, it sets the asyncio event_loop_policy.

This gave me an idea, what if we temporarily restore the event loop policy, get a loop, apply nest_asyncio and then restore the event loop policy.

import nest_asyncio

import asyncio

_cur_event_loop_policy = asyncio.get_event_loop_policy()

asyncio.set_event_loop_policy(asyncio.DefaultEventLoopPolicy())

loop = asyncio.new_event_loop()

nest_asyncio.apply(loop) # type: ignore

asyncio.set_event_loop(loop)

result = loop.run_until_complete(init())

loop.close()

asyncio.set_event_loop_policy(_cur_event_loop_policy)

This seems to do the trick and I am able to call an async function in my FastAPI application main.py before initializing app.

FastAPI and Swagger2

 I recently started working on a web project with Python backend. The hosting was on Google Cloud. 

 We built the POC version of the Python backend using the most popular Python3 framework FastAPI. We deployed it on Cloud Run (public) with Docker image and everything worked as expected. 

 We then moved on to making the Cloud Run private and adding an API gateway infront of it. We saved the http://localhost:8000/openapi.json and ran the gcloud cli to create an API gateway and it failed!

 Looking at the documentation https://cloud.google.com/api-gateway/docs/openapi-overview we realized, Google, for some reason, still only supports OpenAPI 2.0 (Swagger 2.0). Working on AWS for quite some time, I had assumed OpenAPI 3 would be supported.

 We tried figuring out a way to configure FastAPI to generate Swagger 2 spec. But FastAPI only supports OpenAPI 3.x. Many have suggested downloading the openapi.json, passing it through some convertor and then some manual intervention to convert it to Swagger 2. 

• https://github.com/tiangolo/fastapi/issues/1665 | https://github.com/tiangolo/fastapi/discussions/7321
• https://stackoverflow.com/questions/68184751/how-to-generate-swagger-2-0-documentation-for-fastapi
• https://github.com/tiangolo/fastapi/discussions/8235

 Since I wanted to use CI/CD, I didn't want to deal with manual conversion every time developers made some change. So, I referred the FastAPI OpenAPI related code and came up with

Requirements

Python 3.8+
FastAPI 0.79.0+

Installation

$ pip install fastapi_swagger2

Example

from typing import Union
from fastapi import FastAPI
from fastapi_swagger2 import FastAPISwagger2

app = FastAPI()
FastAPISwagger2(app)


@app.get("/")
def read_root():
    return {"Hello": "World"}


@app.get("/items/{item_id}")
def read_item(item_id: int, q: Union[str, None] = None):
    return {"item_id": item_id, "q": q}

This adds following endpoints:

http://localhost:8000/swagger2.json
http://localhost:8000/swagger2/docs
http://localhost:8000/swagger2/redoc

Generate spec for CI/CD

import os

import yaml

from app.main import app

URL = os.environ["CLOUD_RUN_URL"]

app.servers.append(URL)

spec = app.swagger2()

spec['x-google-backend'] = {'address': URL}

print(yaml.dump(spec))

Run GUI applications inside Multipass (Ubuntu) on macOS

Multipass is a nice docker like vm platform from Ubuntu. It can configure the vm with cloud-init so you can prototype your cloud launches locally for free. 

E.g. Launch an instance (by default you get the current Ubuntu LTS)

$ multipass launch --name foo

You can read more about multipass in the docs.

Primarly it does seem like just another alternative to docker. But then I found one nice little difference, not a resource hog unlike docker and you don't run as root.

Currently to run a GUI application which you just want to try without having to clean up the system, most common approach is create a vm with VirtualBox, VMware Fusion or Parallels. But it can be a time consuming task as it installs everything from scratch.

Getting GUI applications to work on docker is possible configuration. But then, atleast on Mac, Docker is a resource hog.

With multipass I was able to get IntelliJ IDEA with following:

Prerequisites:

• Install Multipass
• Install XQuartz. Also configure XAuthLocation.

  echo "XAuthLocation /opt/X11/bin/xauth" >> ~/.ssh/config

Launch a vm with current LTS ubuntu with 4 cores CPU and 4 GB memory

$ multipass launch -n idea -c 4 -m 4G

$ multipass shell idea

$ sudo apt update

$ sudo apt install -y libxrender-dev libxtst6 libxi6 fontconfig

$ cd /tmp

$ curl -L -O https://download.jetbrains.com/idea/ideaIC-2021.1.tar.gz

$ sudo tar -xzf ideaIC-2021.1.tar.gz -C /opt

$ echo 'PATH=/opt/idea-IC-211.6693.111/bin:$PATH' >> ~/.bashrc

$ sudo apt install -y openjdk-11-jdk

You could automate the above with cloud-init. (Currently as of this writing multipass only supports yaml configs)

Now time for the magic part (on the host):

multipass shell internally is just doing a SSH

$ sudo cp '/var/root/Library/Application Support/multipassd/ssh-keys/id_rsa' ~/.ssh/multipass_id_rsa

Save this script to your PATH

$ cat ~/bin/multipass_x_ssh

 

#!/bin/bash

IP=$(multipass info $1 | grep IPv4 | awk '{ print $2 }')

ssh -X -i ~/.ssh/multipass_id_rsa ubuntu@$IP

Now when I want to run idea, all I have to do is:

$ multipass_x_ssh idea

$ idea.sh

pyenv install error on MacOS Big Sur - sendfile invalid

There seems to be some issue going on with pyenv and MacOS Big Sur.

./Modules/posixmodule.c:8320:15: error: implicit declaration of function 'sendfile' is invalid in C99 [-Werror,-Wimplicit-function-declaration]

        ret = sendfile(in, out, offset, &sbytes, &sf, flags);

After referring multiple posts, this is what worked for me.

brew install zlib bzip2

CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" pyenv install 3.5.3 --patch < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)