It has been a while since I started this project to create a modern web service using AWS lambda to fetch and process data from legacy oracle database and it has been very hard for me trying to get node-oracledb(oracle db driver for NodeJS) and mandatory instant client library deployed to lambda using serverless framework. I tried probably the only reletive solution out there called oracledb-for-lambda, which has a precompiled image for instant clienct library. However, it was lacking documentation, not well-maintained and not intended to be used with serverless framework. Since I already have all the function and deployment setup using serverless framework, I found lambda layers would be the best way to use this node-oracledb driver with the precompiled instant client from oracledb-for-lambdaand with the ablility to share with other similar functions with ease.
I would assume you’re already familiar or at least having some knowledge about AWS lambda and serverless framework so I won’t talk too much about neither of them. I have been a big fun of using serverless framework to develop and deploy lambdas and if you’ve never used serverless framework before, you should definitely check it out as it might make your life way easier with lambdas especially for deployment.
Including Library Dependencies in a Layer
You can move runtime dependencies out of your function code by placing them in a layer. Lambda runtimes include paths in the
/optdirectory to ensure that your function code has access to libraries that are included in layers.(AWS layers doc)
Oracle doc says we need to put compiled instant client into /opt/lib and from AWS lambda layers doc, we know that layers will be included in the /opt directory so we will need to put our lib outside the Nodejs folder so that it will be resolved into /opt/lib.Here is my sample lambda layer folder structure with node-oracledb and precompiled instant client lib.
Lambda-Layer-1(version 1)
|
|__lib
| |__libaio.so.1
| |__libclntsh.so.12.1
| |__libclntschcore.so.12.1
| |__libipc1.so
| |__libmql1.so
| |__libnnz12.so
| |__libociicus.so
| |__libons.so
|
|__nodejs
|
|__node_modules
|
|__oracledbAdditionally, you don’t have to worry about changing the default LD_LIBRARY_PATH because lambda default is pointing to /opt/lib:
LD_LIBRARY_PATH: /lib64:/usr/lib64:$LAMBDA_RUNTIME_DIR:$LAMBDA_RUNTIME_DIR/lib:$LAMBDA_TASK_ROOT:$LAMBDA_TASK_ROOT/lib:/opt/libSet up layers in serverless yml
Once we have our layer’s folder structure set up, it is pretty straightforward to set up layers config in serverless.yml. You can just add layers definition and reference them in your functions in serverless.yml. Below is an example and you can reference the serverless documentation for more lambda layers setup guidelines. I am using a Ref here to reference layer defined in the same application(You have to append LambdaLayer to the name of layer you defined when you reference since it’s using cloudformation syntax), you can also use layer’s arn if you already have a layer that it’s not in the same scope, just be sure to also add GetLayerVersion permission to your iamRoleStatements.
#serverless.ymllayers:
oraclelib:
path: layer-dir # required, path to layer contents on disk
name: ${self:provider.stage}-layerName #optional
function:
hello:
handler: handler.hello
layers:
- {Ref: OraclelibLambdaLayer}
Usage in your function
After you set up your layers, you can just require the package in your function as usual without any hassle. Since lambda will look for the same /opt directory that lambda and layers shared, it feels like a magic when you stacking multiple layers together and they just work like a charm. In this case, you just need to require oracledb in the file where your function is defined.
#handler.jsvar oracledb = require('oracledb');
Config webpack if you are using webpack
Either you use babel or typescript, you should consider use serverless-webpack to pack your code so you will have the options to force exclude dependencies from layers as well as running it locally using Serverless-offline.
It’s really important to note that you will need to install dependencies to be used from your layers as dev dependencies for your app. You need to be sure to force exclude them in order to make webpack not complaining when building if you are not using layers for the entire node_modules and modifing your node path.
For example, here is a very simple webpack config snippet from myserverless.yml. I included all the modules for prod build and force exclude the module which I would use from lambda layers. There’s no need to change any set ups you already have in your webpack.config.js,
#serverless.ymlplugins:
- serverless-webpackcustom:
webpack:
webpackConfig: ./webpack.config.js
includeModules: # enable auto-packing of external modules
forceExclude:
- oracledb
Benefit of using lambda layers
The most obvious benefit of using layers is the ability to share same module or function for different functions. In this case, by using lambda layers, rather than include this giant oracle instant client library in every lambda fucntions that requires oracledb, I can simply reference to this layer with only couple lines of code.
Besides, another stunning benefit of using layers is the size of deployment. We all know lambda has a size limit for 50MB after zip and it’s very easy to exceed if you are using some external libraries like full oracle db driver. For instance, here is the size when I try to deploy my layers and my function with precompiled oracle instant client light:
Serverless: Uploading service ***-service.zip file to S3 (5.7 MB)...
Serverless: Uploading service oraclelib.zip file to S3 (28.03 MB)...By separating the large modules into lambda layers, it shrinks down the deployment size for the function itself and opens up more possibilities to share and reuse code for lambda. It’s easy to seperate your common node modules into the 5 different parts( 4 layers and 1 main lambda) with 50mb after zip size each. So in total, if you are crazy enough, you can get 250mb after zip lambda with help of lambda layers. This strategy will not only apply to Node, it will also apply to other languages such as Python, especially if you are using something like numpy or other machine learning library with large dependencies.
Sample oracledb layers starter template
I also created a starter template you can use to bootstrap a serverless app with node-oracledb layers built-in. You will need to modify your db creds and serverless settings to make it work but should be pretty straightforward. You can check it out and clone from here:
Github: https://github.com/jimmdd/node-oracle-layers-serverless
Clone: https://github.com/jimmdd/node-oracle-layers-serverless.gitPlease feel free to leave me messages for questions and please do contribute to the repo on github if you have awesome ideas!