Week 8 Posting - Data Deduplication

Something I have been curious about for a while, and it came up again this week, is data deduplication. For my on-premise data centers, we always run dedupe and compression on the systems to squeeze more out of the raw hardware. I wonder how much cloud service providers can deduplicate data across customers, though. Surely they have to be able to do it at scale to make the business model work in their favor.
Based on an article from Microsoft this week, some deduplication can get as high as 95% efficiency. This is a specific file type for consumers, but do CSPs run the same algorithms? There are also providers, like Box.com that don't even charge by how much data you store with them. It is all about user licenses with unlimited capacity to store data. I think they have to have some way of gaining massive optimizations across users to make that work. It must be about the scale at the end of the day.

Comments

Post a Comment

Popular posts from this blog

Week 4 Posting - Subnetting in the Cloud

Something interesting when looking at subnetting in the cloud is the amount of translations that need to happen in the backend for everything to work. With the numerous customers that a cloud service provider like AWS would be hosting, there would need to be the ability to have completely isolated, but still routable RFC1918 networks sitting beside all their other customers. When looking through the AWS documentation there doesn't seem to be a limitation on what IP addresses can and can't be used, and as I have spun up my own networks it is very flexible. Multiple layers of routing need to happen for this all to work. The customer routing, the datacenter infrastructure routing, AWS internal routing, and their connections to the outside world. Since AWS allows customers to provision their own networks, this has to happen seamlessly in the background using templated provisioning. It would be a wonder to see what this architecture looks like and how AWS can keep up with it.
Read more

Week 6 Posting - Security Appliance in the Cloud

One of the most interesting network accomplishments I have completed in my career is getting a Fortigate firewall appliance provisioned in the cloud. Cloud security has not always been intuitive and there are many vulnerabilities that are exploited because systems are left exposed unintentionally. To overcome this challenge I decided to provision the same type of firewall we used on-premise in the cloud. The challenge was provisioning the cloud appliance. Fortinet had images on their market that could be pulled for the appliance, but in the provisioning phase we needed to apply all the resources needed up front, including cores, memory, interfaces. We did have to do this a couple of times to get it right and figure out the ordering of deployment. Once the appliance was up it was only a matter of getting the Vnet tied to the appliance interfaces and set up routing tables to create defined internal and external networks. After it was completed we could build an IPSec tunnel from the clou...
Read more

Week 5 Posting - VXLAN and Broadcast Security

A newer concept to me is the VXLAN. I have heard of spanning VLANs over layer3 networks, but it is typically an expensive endeavor that has never made sense to me in the past. Using VXLANs where the MAC is inserted at layer 4 instead of layer 2 to create an overlay network seems like it might be a better option where broadcast domains need to be joined. The thing that I don't like doing in my own networks, though, is relying on broadcast or flat network dependent traffic. Usually my experience has been that this dependence is related to poorly developed applications. This might be changing, and it would be interesting to know if companies are using VXLANs more or if security best practices are changing in a way that allows this to be a standard. One of the things I have seen changing a little is better endpoint security, where the network layer isn't the most prominent place to protect systems. EDR has brought a lot of security down to the host. If this trend continues, the mor...
Read more