NimbleBox Deploy
Your token protected API endpoint.
Last Update: 17th September, 2022. Platform images are intentionally not added.
Deploy or nbox.Serving are the live API endpoints that take in requests via HTTP/REST. It is a really general concept since we can not only deploy ML Models into production but esentially be used to run any stateful computation in your workflow.
Quick Start
Here's a quick example on how to make a Deploy using an Operator:
- Start by making a new directory with
mkdir moonshot && cd moonshot - And create a new file
touch baz.py
from nbox import operator
@operator()
def foo(i: float = 4):
# NOTE: json serialisable return
return {"data": i * i}
if __name__ == "__main__":
foo_remote = foo.deploy(type = "serving")
print(foo_remote(10))- Run the code
python3 baz.py
Adding the __name__ == "__main__" guard is critical to ensure there is only one parent process running.
ML Models
Deploying ML models in production is a breeze. Let's just say you want to serve an sklearn model in production. Here's what you are going to do:
- Create a new folder, store the
model.pklpickle file in that folder - Copy the code below and put it
fie.py. Note that instead of using the@operatordecorator we are subclassingOperator. There are two functions you need to define the__remote_init__where you initialise your model andfowardwhere you invoke your model
from nbox import Operator
class AnyModel(Operator):
def __remote_init__(self):
self.model = pickle.load(open("model.pkl", "rb"))
def forward(self, x: int) -> int:
return self.model.predict(x)
if __name__ == "__main__":
foo_remote = AnyModel.deploy(type = "serving")
print(foo_remote(10))The current deployment will be terminated when the new upload is complete. nbox will convert the model to a fastAPI endpoint and you can:
- HTTP/REST command at
https://api.nimblebox.ai/{depl_id}/forward_rest - Opertor RPC using
Operator.from_serving(depl_id), which is what happens in the above code.
Advanced
If you think about it any class in Python language is either used as a namespace or because it carries some state that will either be modified or used in some other part of your flow. Thus classes are by default hosted as Servings when deployed. Here's a quick model
from nbox import operator
@operator()
class StatefulCloudObject(Operator):
def __init__(self):
self.x = 4
def inc(self):
self.x += 1
def dec(self):
return self.x
if __name__ == "__main__":
cloud_state = StatefulCloudObject.deploy() # classes by default are deployed as serving
print(cloud_state.x) # retrieve objects over the internet
cloud_state.inc()
print(cloud_state.x)
cloud_state.dec()
print(cloud_state.x)User must be ensure that their code does not lead to race conditions.
Live Endpoint Monitoring (WIP)
Simply deploying the models is not enough and you need far more insight into what your model is doing, for this we have Live Enpoint monitoring that connects using nbox.lmao.LMAOAsgiMiddleware and logs all the information. You can optionally configure it to detect drifts in your endpoints really easily.