Linux will always try to use ALL the RAM. That does not mean you will run into any issues if you do not have much free RAM. Only if “Used“ were to fill up all the RAM, you should be worried.
Cache and buffers are just Linux trying to improve storage speeds by keeping things in RAM. Restarting because all RAM is used for cache and buffers would actually not help it, because it would then again need load all it decided to buffer and cache in RAM from storage.