After a nice vacation, I come back with the solution for quiz-7.
Have a look at the quiz to understand the problem.

Quiz Review

As indicated in the quiz, the network administrator discovered an access switch without MLS QOS (while all the other switches in the network were configured with it) and he took the initiative to enable it without thinking of the consequences.
As a result, from the moment he enabled MLS QOS, the switch started re-writing the DSCP bits of all packets and since no other QOS-related configuration was performed, the switch reset all QOS values to zero. This can be easily spotted with the show mls qos command: 

Access-4(config)#mls qos 
Access-4(config)#^Z
Access-4#
Access-4#sh mls qos
QoS is enabled
QoS ip packet dscp rewrite is enabled

Since packets will have the DSCP reset to zero, the rest of the network will treat them in the best-effort class, which explains the latency problems.
The small trick hidden in the quiz is the question "Why everything was ok before his actions?". As most of you replied within the comments section, the answer is simple: with MLS QOS disabled, the switch does not change anything because there's no concept of trust or untrust, so packets sent with various DSCP/COS/IPPrec settings were forwarded by the access switch and treated accordingly by all other devices in the network (of course, this means that marking was performed by the application running on the end host).

QOS Review

Now, let's step back a little and review some things about QOS:

First of all, let's see various headers and the QOS bits within them (marked with red color):
QOS-bits

Classification

To apply a different treatment to different traffic, the switch must distinguish packets from one another in a process called classification that results in generating a QOS label that will identify further actions which will be performed with the packet.

classification

  • The COS-to-DSCP and the IPPrec-to-DSCP maps have default values that may or may not be appropriate for your network.
  • The default DSCP-to-DSCP-mutation map and the default policed-DSCP map are null maps; they bind an incoming DSCP value to the same DSCP value.
  • The DSCP-to-DSCP-mutation map is the only map you apply to a specific port. All other maps apply to the entire switch.

Policing and Marking

As packets move through the switch, you can force them to comply with configured policies or profiles - policing determines if the a packet is in or out of profile (based on bandwidth/resource usage limits) and passes the result to marking process that will read configuration and take actions, such as:

  • pass it further without modification
  • drop down the QOS label
  • drop it.

If the packet is out of profile and a drop down action is speficied, the switch will use the policed-DSCP map to generate a new QOS label.

Queueing and Scheduling

Since the total inbound bandwidth of all ports can exceed the bandwidth of the internal ring, ingress queues are used before packets are forwarded into the switch fabric. Later on, outbound queues are located/used to send them onto an egress port.

In a future post, I'll try to come with more explanations about queueing and scheduling, as this topic is rather large.

Quiz Solutions

In the end, let's get back to our quiz and say that the network administrator did not do a mistake by enabling MLS QOS, as one of the QOS design rules is to clasify and mark data applications as close to their sources as possible. The mistake was his lack of knowledge about the DSCP re-write.
Hosts and servers are capable of marking the COS and DSCP values but it is a question of whether to trust them or not. Without MLS QOS, the switch does not interfere with these markings.
While enabling MLS QOS, the administrator needs to define the trust boundaries. Some devices can be trusted (fully or partilly/conditionally - see IP Phones) or untrusted. The access switches are the closest to the endpoints, so here is where you usually define your trust boundaries.

Solutions to the quiz:

  • trust the QOS markings received from the application running on the end-host (or use DSCP-to-DSCP mutation maps)
  • enable DSCP Transparency Mode using the no mls qos rewrite ip dscp command which will instruct the switch not to modify the DSCP values of the incomming packets
  • use QOS ACLs to match the desired traffic generated by the specific application and apply the DSCP values to it

Thanks everyone for your comments in the quiz !