How can I be more specific

Resolution and accuracy


Whether or not a measuring device is suitable for a measuring task depends heavily on the requirements that the measuring device must meet in order to carry out a measurement that is sufficiently precise for the user. An important requirement is the resolution and accuracy of the measurement. The measuring accuracy of a measuring device can often be found under the term “basic accuracy”. The value has the unit [%] and indicates the maximum percentage by which the result measured by the device can deviate from the physically correct value. A basic accuracy of ± 0.05% means that the measured value deviates by a maximum of + 0.05% or -0.05% from the physically correct value. The basic accuracy specified by the manufacturer usually reflects the smallest possible deviation, which can be increased under certain circumstances by changing parameters such as the measuring frequency or the measuring speed.


Another important factor for a qualitative measurement is the resolution. It indicates the increments in which measured values ​​can be displayed. The higher the resolution of a measuring device, the more precisely a value can be represented. In other words, the higher the resolution, the more different values ​​can be displayed in this measuring range. For example, if the measuring range is 0Ω - 1000Ω with a resolution of 1Ω, 1000 different measured values ​​can be displayed. However, if the resistance value is to be measured with an accuracy of 0.5 Ω, a measuring device with a resolution of at least 0.5Ω is required. The more requirements on the measuring device are known in advance, the better the measuring device can be matched to the application.

Example 1: "Calculation of the measuring device"

TARGET values: You want to measure a resistance between 0.01 Ω and 0.05 Ω. Your tolerance should be +/- 5%. Then the following calculation results:10 mΩ (0.01 Ω) * 5% = 0.5 mΩ

ACTUAL values: e.g. resistance meter DU5010 with a measuring range of 200 mΩ: 0.01 mΩ - 199.99 mΩ. The following calculation results:200 mΩ * 0.1% = 0.2 mΩ

Since the 0.2 mΩ change is smaller than 0.5 mΩ, the accuracy of the DU5010 ohmmeter would be completely sufficient.

Example 2: "Calculation of resolution and accuracy"

Resistances of 5 Ω should be measured. The maximum deviation of the resistances may be 0.1 Ω. The minimum required resolution of 0.1 Ω results from the maximum deviation that is to be measured. The deviation of 0.1 Ω then corresponds to the following percentage deviation:(0.1 Ω / 5 Ω) * 100 = 2%

A measuring device is therefore required that has a basic accuracy of less than 2%.
This makes 2 requirements clear:

Resolution ≤ 0.1 Ω

Basic accuracy ≤ 2%

" Back