# Scaling up BERT-like model Inference on modern CPU - Part 1

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fscaling-up-bert-like-model-inference-on-modern-cpu-part-1%2F5653%3E%5C
**Category:** Intermediate\
**Created:** [April 21, 2021, 9:31am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fscaling-up-bert-like-model-inference-on-modern-cpu-part-1%2F5653 "2021-04-21T09:31:44Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Matthieu](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/matthieu/32/3295_2.png) [@Matthieu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMatthieu)%5C
**Post date:** [April 21, 2021, 9:31am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fscaling-up-bert-like-model-inference-on-modern-cpu-part-1%2F5653%2F1 "2021-04-21T09:31:44Z")

</div>

Hi community,

I have come through the nice article by @mfuntowicz : [Hugging Face – The AI community building the future.](https://huggingface.co/blog/bert-cpu-scaling-part-1)

It sounds really interesting how easily you can benchmark your BERT transformer model with CLI and [Facebook AI & Research’s Hydra configuration library](https://hydra.cc/).

Is it possible however to easily test it on cloud services as AWS and how to deploy it?

Thanks!

---

<div class="post-metadata">

**Author:** ![mfuntowicz](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mfuntowicz/32/3514_2.png) [@mfuntowicz](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmfuntowicz)%5C
**Post date:** [April 21, 2021, 1:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fscaling-up-bert-like-model-inference-on-modern-cpu-part-1%2F5653%2F2 "2021-04-21T13:37:23Z")

</div>

Hey @Matthieu,

Thanks for reading and posting here 🙂.

Indeed, everything in the blog was run in AWS (c5.metal) instance(s).  
The way I’m currently using it:

- `git clone https://github.com/huggingface/tune`
- `cd tune`
- `pip install -r requirements.txt`
- `export PYTHONPATH=src`
- `python src/main.py --multirun backend=pytorch batch=1 sequence_length=128,256,512`

The overall framework is quite new and I’ll be improving the UX in the coming days, sorry for this user experience 😇

Morgan 🤗

---

<div class="post-metadata">

**Author:** ![Matthieu](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/matthieu/32/3295_2.png) [@Matthieu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMatthieu)%5C
**Post date:** [April 21, 2021, 1:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fscaling-up-bert-like-model-inference-on-modern-cpu-part-1%2F5653%2F3 "2021-04-21T13:58:28Z")

</div>

Hi @mfuntowicz

Thanks again for the great article.

> Indeed, everything in the blog was run in AWS (c5.metal) instance(s).  
> The way I’m currently using it:
> 
> - `git clone https://github.com/huggingface/tune`
> - `cd tune`
> - `pip install -r requirements.txt`
> - `export PYTHONPATH=src`
> - `python src/main.py --multirun backend=pytorch batch=1 sequence_length=128,256,512`

1. So you ran these command lines directly from the c5.metal instance? You didn’t need to install before on it a docker image with OS and pip/python packages?
2. With this overall framework you can simulate different hardware parameters on the latency/throughput outputs. But, generally transformers are encaspulated within a docker image providing an API before deployment on cloud services. How could this benchmark could simulate real latency/throughput of the docker image deployed?

Matthieu

---

<div class="post-metadata">

**Author:** ![mfuntowicz](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mfuntowicz/32/3514_2.png) [@mfuntowicz](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmfuntowicz)%5C
**Post date:** [April 22, 2021, 1:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fscaling-up-bert-like-model-inference-on-modern-cpu-part-1%2F5653%2F4 "2021-04-22T13:54:48Z")

</div>

> 1. So you ran these command lines directly from the c5.metal instance? You didn’t need to install before on it a docker image with OS and pip/python packages?

Yes, exactly.

> 1. With this overall framework you can simulate different hardware parameters on the latency/throughput outputs. But, generally transformers are encaspulated within a docker image providing an API before deployment on cloud services. How could this benchmark could simulate real latency/throughput of the docker image deployed?

That’s an interesting point. We do not provide a testbed for integrated solution (_yet?_). Still, all the knobs discussed in this first part and the ones comming in the second part are leverageable within a container and should see the same performances benefits highlighted in the blog posts.

Of course, it doesn’t simulate the latency overhead of a web server handling incoming requests and/or dynamic batching as would Nvidia Triton do for instance.

Hope it helps,  
Morgan
